Add authentication with ServiceAccountToken

[yanniszark]

Problem Statement

Clients in various namespaces (e.g., Notebooks) need to access the Pipelines API. However, there is currently no way for these clients to authenticate to the Pipelines API:
#4440
#4733
In-cluster clients need a way to authenticate to the KFP API Server.

Proposed Solution

The correct way to do this is by using audience-scoped ServiceAccountTokens. In Arrikto's Kubeflow distribution, we have been successfully using this method for a long time, in numerous customer environments. We want to upstream this solution so the whole community can benefit as well, since we see this is an issue many users bump into.
Changes need to happen in 2 places:

• API Server, which needs to support authentication with ServiceAccountToken.
• KFP Client, to better support this authentication method.

/assign @yanniszark
cc @Bobgy

[Bobgy]

Thank you for the proposal! I'd love to see it getting it upstreamed too. It's a common request in #4440.

[elikatsis]

Hello!

I'll provide an update here as I'll be pushing a PR covering the backend part very soon.

As mentioned in the first comment, we are adding a new authentication method: authentication using ServiceAccountTokens.
For this, we need the clients to put ServiceAccountTokens in requests and the backend (KFP API server) to retrieve them and authenticate the requests.

How will this ServiceAccountToken find its way in the requests?

1. The client finds a proper ServiceAccountToken (more on this later on)
2. It adds an Authorization: Bearer <token> header in all requests

What does the authentication cycle of the backend look like?

1. We will extend the authentication mechanisms of the KFP API server with one more authenticator [and we will make the available authenticators extendable]
2. Every request will pass through all available authenticators (currently, Kubeflow-UserID header and ServiceAccountToken) until one succeeds.
   Then, that is, if one succeeds, authentication succeeds.
   Otherwise, that is, if all authenticators have failed, the request is considered unauthenticated.

How does the ServiceAccountToken authenticator work?

1. The KFP API server creates a TokenReview using the ServiceAccountToken retrieved from the requests bearer token header and some expected audience (for the KFP case, this can be ml-pipeline)
2. Kubernetes responds (with the TokenReviewStatus) whether the token is associated with a known user and with what audience
3. The KFP API server verifies that ml-pipeline is in the audience specified in the Kubernetes response
4. The KFP API server considers the request authenticated and assumes the user specified by Kubernetes in its response

Useful links:

• https://kubernetes.io/docs/reference/access-authn-authz/authentication/
• https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.16/#tokenreview-v1-authentication-k8s-io

How does the client find a ServiceAccountToken to use?

Kubernetes has built-in ways to project tokens with specific audience for the ServiceAccount of a pod.
Each container of a pod mounts the token similarly to how it would mount some volume.
The kubelet generates a token and stores it in a file. Then, to retrieve the token, it's just a matter of reading this file.

The KFP client should have a seamless way to

1. retrieve the path where the token is mounted,
2. read it, and
3. use it in request headers.

The token has an expiration time, however the kubelet makes sure to refresh this token before it expires.
So, finally, the client should re-read the token every now and then.

This last part is also relevant to the discussion of #4683

Useful links:

• https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection

[davidspek]

@elikatsis @yanniszark If any help is needed so this can be added for 1.3 please let me know. I think a large portion of the community has been waiting for this a while now and it would be great to have it included in 1.3.

[elikatsis]

/assign @elikatsis

[elikatsis]

@davidspek thanks for volunteering!
We actually have the code ready for a PR, and we've extensively tested it in our deployments.
I believe it would be really helpful if had the time to test the PRs (backend & client)!

Before I open the client PR I'll present some implementation details (we've described an overview in the comment above)

As mentioned in #4683 (comment), we want to have a generic way to provide credentials to the client. We will be using a TokenCredentials abstract class for this and we will be making use of a very interesting built-in Kubernetes Configuration functionality: auth_settings. [Obviously, we use a Configuration in our client (source).]

Requirements

1. We want some credentials to find their way in request headers and, more specifically, in the Authorization: Bearer <token> header.
2. Also, we need a way to refresh the token before making an API call (as mentioned in the comment above, when projecting service account tokens for pods, the kubelet refreshes them every now and then, so a client needs to read the token often)

Information about the Kubernetes Configuration object

1. A Kubernetes Configuration, based on its attributes, it may hold some BearerToken authentication settings (source)
2. Before making an API call it updates the request using these settings (source). Based on these settings, it may populate the request with:
   • cookies,
   • headers, or
   • queries.

[Expanding (1)] As shown in this source, by providing a Configuration.api_key["authorization"] we can add a BearerToken auth setting which:

1. adds a header to the request (source)
2. the header name is authorization (source)
3. the header value is retrieved using the get_api_key_with_prefix() method (source)

[Expanding (3)] The get_api_key_with_prefix() method (source)

1. Eventually returns self.api_key["some-key"] with a desired prefix if self.api_key_prefix["some-key"] is set
2. Note that before running any of this, it executes the refresh_api_key_hook() method if it is defined ❗

[Expanding (2)] The refresh_api_key_hook() method runs before every request. And, as its name suggests, it's a neat way to refresh the api keys!

Conclusions

To sum up, what we need to do is:

1. populate our config.api_key["authorization"] = token,
2. populate our config.api_key_prefix["authorization"] = "Bearer", and
3. provide our config.refresh_api_key_hook with a function that updated config.api_key["authorization"].

So, for this case (authentication with ServiceAccountTokens), we need to

1. Read the contents of a specific file in the container's file system (projected service account tokens are essentially volumes mounted on pods). This is the token
2. Use a method that reads and returns the contents of this file as the refresh_api_key_hook

Design decisions

1. We will create a subclass of TokenReview named ServiceAccountTokenVolumeCredentials
2. The class constructor will be expecting a path pointing to the file where the token is stored
3. If the user doesn't provide a path, the constructor will look for an environment setting: the value of the environment variable ML_PIPELINE_SA_TOKEN_PATH
4. If the user doesn't provide a path and the environment variable is not set, the constructor will fall back to reading the path /var/run/secrets/ml-pipeline/token
5. The Client constructor will be expecting a credentials argument and manipulate it accordingly
6. If no credentials are provided and the client detects it is running inside a pod, it will attempt to use a ServiceAccountTokenVolumeCredentials.

How to set up the pod to authenticate against KFP

We (Arrikto) have been using a PodDefault that configures the pod to authenticate against KFP based on the aforementioned design.
Here follows the PodDefault, it essentially describes all that we need to supplement the pod definition with:

apiVersion: kubeflow.org/v1alpha1
kind: PodDefault
metadata:
  name: access-ml-pipeline
spec:
  desc: Allow access to Kubeflow Pipelines
  selector:
    matchLabels:
      access-ml-pipeline: "true"
  volumeMounts:
  - mountPath: /var/run/secrets/ml-pipeline
    name: volume-ml-pipeline-token
    readOnly: true
  volumes:
  - name: volume-ml-pipeline-token
    projected:
      sources:
      - serviceAccountToken:
          path: token
          expirationSeconds: 7200
          audience: ml-pipeline
  env:
  - name: ML_PIPELINE_SA_TOKEN_PATH
    value: /var/run/secrets/ml-pipeline/token  # this is dependent on the volume mount path and SAT path

[davidspek]

@elikatsis Thanks for the detailed post. I will look at it more closely tomorrow and do my best to help test the PRs.

[elikatsis]

@Bobgy, @davidspek I've opened two PRs 🎉

1. Backend: feat(backend): Support authentication with ServiceAccountTokens. Part of #5138 #5286
2. Client: feat(sdk): Introduce the token credentials interface. Part of #4683 #5287