support separate pipeline for each namespace

[yaliqin]

EDIT from @Bobgy: This issue got 47 upvotes when requesting user feedback: #1223 (comment).

Proposed change

Currently with 1.0.2 version, Kubeflow Deployment with kfctl_k8s_istio shares pipelines for all namespaces defined in user profile. Separate pipeline support for each namespace is needed because the multi-user notebook server separation is already supported. It is natural to support pipeline separation.

Alternative options

NA

Who would use this feature?

A lot of enterprises will benefit from this feature as allocating different namespaces to different teams is a common practice in Kubernetes and resources in existing namespaces can be effectively used.

Suggest a solution

NA

[Bobgy]

FYI, in existing to be released KF 1.1, pipeline runs are already in user namespaces.

but the static pipeline yaml files are not separated. Do you want this additionally?

[Bobgy]

Related to #1223

[Bobgy]

There are more clarification in #1223 (comment).

If you agree with the proposal in #1223 (comment), can you rephrase your description to just focus on pipeline resource for this issue.

[yaliqin]

I think the pipeline yaml files should be supported also for E2E separation. Thanks. Already upvote in #1223 .

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Bobgy]

/frozen

[Bobgy]

Sorry, I'm kind of oversubscribed because of extra Kubeflow duties recently, so I may not be able to take this issue soon.

Leaving my previous thoughts about this:

For use-case context: some users want full separation of pipelines, and some want no separation. The distinction is mostly organization culture and I think both requests are valid.

So I think a MVP UX I can imagine that satisfies both is to:

1. pipelines in DB should have a namespace column
   • if namespace == '' (empty string), then it's in the shared space
   • if namespace is not empty, it's in that namespace
2. all requests to get pipelines can specify namespace='' to query only shared pipelines
3. or specify namespace='XXX' to find pipelines in a namespace + shared pipelines
4. We need to add a config option that disables shared pipelines for organizations that definitely do not want shared pipelines
5. We need to add a KFP UI option to switch between uploading a pipeline to shared space or current namespace.

Benefits:

1. The same backend logic seems to be able to support both 'all shared' or 'all separated' use cases.
2. There's no DB migration needed, this will be a seamless transition from older versions of KFP.

Probably there are cases I haven't fully thought through. I'd suggest anyone who's willing to push this issue forward to do the following:

1. try to discuss and converge on a set of CUJs (critical user journey) that works for everyone, note we should probably start with MVPs that address immediate concerns, but make sure we can head towards the direction that all valid use-cases can be supported.
2. after consensus on the UX, discuss with maintainers a rough design of the implementation
3. contribute PRs to implement this feature in smaller self-contained steps