[Question] How to setup a runtime configuration?

[augustovictor]

Could not find any specific doc on what value to put under Kubeflow Pipelines API Endpoint when defining a Runtime Configuration.

So far I have kubeflow running on eks (aws).

What I'm trying to accomplish is a simple elyra's pipeline executed on kubeflow pipelines.

However I cannot figure out what value to put on Kubeflow Pipelines API Endpoint.

Further info

• Elyra's image used for jupyter server configuration: elyra/kf-notebook:3.9.0

What i've tried so far

Getting the ip of ml-pipeline-ui service in k8s and setting Kubeflow Pipelines API Endpoint with it.

Sorry if I missed some obvious steps...

[ptitzler]

Where is your JupyterLab/Elyra running? In the cluster where Kubeflow is running on on a separate machine, e.g. your laptop?

[augustovictor]

Where is your JupyterLab/Elyra running? In the cluster where Kubeflow is running on on a separate machine, e.g. your laptop?

@ptitzler In the kubeflow cluster

[ptitzler]

Thank you! In that case you need to use the KUBERNETES_SERVICE_ACCOUNT_TOKEN authentication type and configure your environment as described in section 'Full Kubeflow (from inside cluster)' of this doc.

As API_ENDPOINT specify http://ml-pipeline-ui.<namespace>:<port>/pipeline, replacing <namespace> with the namespace where Kubeflow is deployed (likely kubeflow [1]) and <port> with the ml-pipeline-ui service's port (likely 80).

[1] to confirm the namespace you can run kubectl get services ml-pipeline-ui -n kubeflow. If this does not return a match try other namespaces.

Note that the links in the Elyra UI, e.g.

won't work because your browser is running on a machine that won't be able to resolve the host name, since it's an in-cluster URL. If you do want that to work you'll need to expose the service to the public and also specify the Public Kubeflow Pipelines API endpoint in your runtime configuration.

One more clarification please. Are you running JupyterLab/Elyra as a Kubeflow Notebook server?

[augustovictor]

Yup. I'm running JupterLab/Elyra as a Kubeflow Notebook server.
I'll try what you've suggested then I get back to you.

I got a bit confused when checking the 'Full Kubeflow (from inside cluster)' section under this page.

It has:

1. The python file that fetches credentials from KF_PIPELINES_SA_TOKEN_PATH and creates a connection;
2. ServiceAccount token volume mount;
3. ServiceAccount token volume injection with PodDefault;

Should I follow all 3?

Thanks =D

[augustovictor]

@ptitzler I got part of it to work

First Applied PodDefault;
Then criated a new jupyterlab server and selected Allow access to kubeflow pipelines in the Configurations section.
When it started I created a pipeline runtime config with the following values:

Then got this message:

The logs in my notebook pod (k -n kubeflow-user-example-com logs victor-elyra-0 -f) says:

ERROR:root:Failed to get healthz info attempt 5 of 5.
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/kfp/_client.py", line 411, in get_kfp_healthz
    response = self._healthz_api.get_healthz()
  File "/opt/conda/lib/python3.8/site-packages/kfp_server_api/api/healthz_service_api.py", line 63, in get_healthz
    return self.get_healthz_with_http_info(**kwargs)  # noqa: E501
  File "/opt/conda/lib/python3.8/site-packages/kfp_server_api/api/healthz_service_api.py", line 134, in get_healthz_with_http_info
    return self.api_client.call_api(
  File "/opt/conda/lib/python3.8/site-packages/kfp_server_api/api_client.py", line 364, in call_api
    return self.__call_api(resource_path, method,
  File "/opt/conda/lib/python3.8/site-packages/kfp_server_api/api_client.py", line 188, in __call_api
    raise e
  File "/opt/conda/lib/python3.8/site-packages/kfp_server_api/api_client.py", line 181, in __call_api
    response_data = self.request(
  File "/opt/conda/lib/python3.8/site-packages/kfp_server_api/api_client.py", line 389, in request
    return self.rest_client.GET(url,
  File "/opt/conda/lib/python3.8/site-packages/kfp_server_api/rest.py", line 230, in GET
    return self.request("GET", url,
  File "/opt/conda/lib/python3.8/site-packages/kfp_server_api/rest.py", line 224, in request
    raise ApiException(http_resp=r)
kfp_server_api.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'content-length': '19', 'content-type': 'text/plain', 'date': 'Tue, 18 Oct 2022 11:09:29 GMT', 'server': 'envoy', 'x-envoy-upstream-service-time': '1'})
HTTP response body: RBAC: access denied

---

Then i tried adding a roleBinding between the ServiceAccount default-editor under my user's namespace, and ml-pipeline-ui (as described here)

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: allow-namespace-2-kubeflow-edit
  ## this RoleBinding is in `namespace-1`, because it grants access to `namespace-1`
  namespace: kubeflow
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubeflow-edit
subjects:
  - kind: ServiceAccount
    name: default-editor
    ## the ServiceAccount lives in `namespace-2`
    namespace: kubeflow-user-example-com

However the error still persists.
What am I missing?

Thanks

[ptitzler]

@thesuperzapper do you have any insights to share that might help in this case? I don't think this is an Elyra specific configuration issue.

[ozen]

@augustovictor I had a similar issue that was solved by changing the api endpoint url: #3092

[ptitzler]

See previous comment. Please re-open if this is still an issue.

[gioargyr]

Hello all,

I am at the same point that @augustovictor was:
Failed to initialize kfp.Client()against: 'http://ml-pipeline-ui.kubeflow:80/pipeline'
The reason for this error is:
TimeoutError: Failed getting healthz endpoint after 5 attempts.

Do you have a specific solution for that?
fyi: changing the api endpoint url didn't help.