Can not find experiments in frontend if namespace is not kubeflow

[Jeffwan]

/kind bug

What steps did you take and what happened:
I am new to katib and not sure if this is an issue.

Based on my finding, I notice if I submit experiments in different namespaces other than kubeflow. There're two problems

1. Can not find experiments on UI
2. Sometimes experiment never get done. Seems pod are done but experiment status are not updated. Checked status using command kubectl get experiments

What did you expect to happen:
We want every individual user create experiment in their own namespace.

Environment:

• Kubeflow version: 0.6.2
• Kubernetes version: (use kubectl version): 1.13.8

[SatyaNarayan1]

facing the same issue kubeflow version 0.6.2. Cant see any thing in experiments UI also experiments/trials in running state.

[johnugeorge]

Yes. This is a known issue. This issue will be solved in upcoming 0.7 release

[SatyaNarayan1]

@johnugeorge is there any earlier version in which experiment UI is working?

[johnugeorge]

I was referring to issue where Katib job and services are in different namespaces:

Related: #343

[johnugeorge]

@SatyaNarayan1 Do you have them in different namespaces?

[SatyaNarayan1]

@johnugeorge I tried both approach

1. Keeping in same namespace "kubeflow"
2. Keeping in two different namespace
   in both cases experiment UI is blank

[gaocegege]

So, current implementation is such that we do not provide namespace, it is uses the namespace of the deployment (which is KF)

From @johnugeorge

[Jeffwan]

@gaocegege @johnugeorge

New to Katib project and do we have an estimate on work to support different namespace. Since most important components are moved to multi-tenancy mode, I hope users can submit their katib experiment in their own namespace. Otherwise, user may not have permission on kubeflow in most of the enterprise envs. This should be a high priority improvement.

[gaocegege]

Hi @Jeffwan
/cc @johnugeorge

I think we are going to give katib cluster scoped access and show all the experiments in all namespaces.

And I will work on it. Does it work for you?

[gaocegege]

/assign

[Jeffwan]

@gaocegege That makes sense. Show all experiments is good enough now.

[gaocegege]

/area front-end