[0.15-rc.1] 401 error: user header not present

[gkcalat]

/kind bug

What steps did you take and what happened:

1. Deployed katib as a part of Kubeflow on Google Cloud.
2. Clicked on Experiments (AutoML) menu item in the centraldashboard UI
3. Request failed with 401 error (returned from here?)
4. Replacing v0.15.0-rc.1 with 0.14.0 resolves the issue

What did you expect to happen:

200 response

Anything else you would like to add:

Deployment:

1. kustomize build -o ./build here
2. kubectl apply -f ./build

Request:

Request URL: https:ENDPOINT/katib/fetch_experiments/?namespace=user-namespace
Request Method: GET
Status Code: 401
Remote Address: ...
Referrer Policy: strict-origin-when-cross-origin
e-time: 2

Request headers:
:authority:
:method: GET
:path: /katib/fetch_experiments/?namespace=user-namespace
:scheme: https
accept: application/json, text/plain, /
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9,ru;q=0.8
cookie:
referer: https://ENDPOINT/katib/
sec-ch-ua: "Google Chrome";v="111", "Not(A:Brand";v="8", "Chromium";v="111"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "macOS"
sec-fetch-dest: empty
sec-fetch-mode: cors
sec-fetch-site: same-origin
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36

Payload:
namespace:user-namespace

Response:
user header not present

Response headers:
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
content-length: 24
content-type: text/plain; charset=utf-8
date: Thu, 16 Mar 2023 21:43:07 GMT
server: istio-envoy
via: 1.1 google
x-content-type-options: nosniff
x-envoy-upstream-servic

The v0.14 request looks as follows:

Request in v0.14:
Request URL: https://ENDPOINT/katib/fetch_experiments/
Request Method: GET
Status Code: 200
Remote Address: ...
Referrer Policy: strict-origin-when-cross-origin

Request headers:
:authority: ENDPOINT
:method: GET
:path: /katib/fetch_experiments/
:scheme: https
accept: application/json, text/plain, /
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9,ru;q=0.8
cookie:
referer: https://ENDPOINT/katib/
sec-ch-ua: "Google Chrome";v="111", "Not(A:Brand";v="8", "Chromium";v="111"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "macOS"
sec-fetch-dest: empty
sec-fetch-mode: cors
sec-fetch-site: same-origin
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36

Payload:

Response:
[{"name":"mnist-e2e","namespace":user-namespace,"type":"hp","status":"Succeeded","startTime":"2023-03-15T18:19:29Z","completionTime":"2023-03-15T18:21:50Z","conditions":[{"type":"Created","status":"True","reason":"ExperimentCreated","message":"Experiment is created","lastUpdateTime":"2023-03-15T18:19:29Z","lastTransitionTime":"2023-03-15T18:19:29Z"},{"type":"Running","status":"False","reason":"ExperimentRunning","message":"Experiment is running","lastUpdateTime":"2023-03-15T18:21:50Z","lastTransitionTime":"2023-03-15T18:21:50Z"},{"type":"Succeeded","status":"True","reason":"ExperimentMaxTrialsReached","message":"Experiment has succeeded because max trial count has reached","lastUpdateTime":"2023-03-15T18:21:50Z","lastTransitionTime":"2023-03-15T18:21:50Z"}],"currentOptimalTrial":{"bestTrialName":"mnist-e2e-ktxfptcc","parameterAssignments":[{"name":"learning_rate","value":"0.036678390805863215"},{"name":"batch_size","value":"97"}],"observation":{"metrics":[{"name":"loss","min":"0.2380017","max":"2.327921","latest":"0.2380017"}]}},"succeededTrialList":["mnist-e2e-4fqn265r","mnist-e2e-ktxfptcc","mnist-e2e-qvmwzpwx","mnist-e2e-24jd787v","mnist-e2e-l4qj6dh2"],"trials":5,"trialsSucceeded":5}]

Response headers:
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
content-length: 1198
content-type: text/plain; charset=utf-8
date: Thu, 16 Mar 2023 22:05:42 GMT
server: istio-envoy
via: 1.1 google
x-envoy-upstream-service-time: 12

Environment:

• Katib version (check the Katib controller image version): v0.15.0-rc.1
• Kubernetes version: (kubectl version): GKE 1.23 and GKE 1.24
• OS (uname -a): linux

---

Impacted by this bug? Give it a 👍 We prioritize the issues with the most 👍

[tenzen-y]

@gkcalat Thank you for reporting this issue!

@kimwnasptd @orfeas-k @elenzio9 Can you check this issue?

[kimwnasptd]

Hey @gkcalat, indeed there's a change in this release related to authnz and thus the error this time compared to the past.

Specifically we worked with @apo-ger on adding authorization checks in the Katib web app in #1983 and #2041. So in this release the web app will expect a header with the user identity, similar to the rest of the web apps.

I think the issue is that

1. by default this header is expected to be kubeflow-userid https://github.com/kubeflow/katib/blob/master/pkg/new-ui/v1beta1/authzn.go#L19
2. I'm guessing you are using IAP on Google Cloud for authn, which adds the identity in a different header in requests

Based on the above I'd suggest one of the following options:

1. (Recommended) Update the Katib web app's manifests to set this ENV var appropriately. I'm guessing you must have already configured this for the jupyter-web-app. You need to set the same env var
2. Disable the authnz altogether https://github.com/kubeflow/katib/blob/master/manifests/v1beta1/installs/katib-with-kubeflow/patches/enable-ui-authz-checks.yaml

[gkcalat]

Hi @kimwnasptd and @tenzen-y.

Thank you for quickly responding to the issue! Do you think this needs to be mentioned in breaking changes?

[tenzen-y]

Do you think this needs to be mentioned in breaking changes?

+1
wdyt? @andreyvelich @johnugeorge

[andreyvelich]

Sure, @kimwnasptd @apo-ger Please can you create PR to update the CHANGELOG with this Breaking Change for Katib 0.15 ?