Uber Issue: Manifests WG Steps for release 1.3

[yanniszark]

Intro

I know that many are eager to track the progress of the manifests working group,
so I'm opening this issue to document the plan, as was presented in the
wg-manifests meeting on 2021/02/05
(recording).

More details about the mission of wg-manifests can be found in its charter:
https://github.com/kubeflow/community/blob/master/wg-manifests/charter.md

Plan

This plan will be updated regularly.

Step 1: Migrate Manifests to a new Folder Structure

Currently, the manifests repo is a flat namespace with everything mangled.
We plan to reorganize the folder structure into sections that will reflect the
following properties:

• Applications with their source code in upstream KF maintained by Kubeflow WGs (e.g., notebook-controller)
• Common services (Dex, Istio, Cert-Manager, KNative), maintained by the
  Manifests WG.
• Applications contributed by community members, not owned by any Kubeflow WG.
• Distribution-specific manifests (kfdef, stacks, aws, gcp, etc.). The goal is
  for this folder to become empty in subsequent releases.

Issue: #1738

Step 2: Consolidate manifests development upstream

Consolidate manifests development upstream. Move all manifests from
kubeflow/manifests to their respective upstream repos.

To give an example from the notebook-controller, the diagram shows the current
situation. Manifests are in two repos, blurring separation of responsibilities
and making things harder for application developers.

Instead, we will copy all manifests from the manifests repo back to each
upstream's repo. From there, they will be imported in the manifests repo. The
following diagram presents the desired state:

Issue: #1740

Step 3: Upgrade Istio

The manifests working group is responsible for certain common services (Dex,
Istio, Cert-Manager, KNative). Out of those, we plan to provide a newer version
of Istio, to leverage the new AuthorizationPolicy API that will be supported
across the board in 1.3.

The work to refactor every component to use Istio AuthorizationPolicy is tracked
in kubeflow/kubeflow#5313 (comment).

Issue: #1753

Step 4: Consolidate manifests with multiple versions into 1

Some manifests (e.g., notebooks controller, katib) have multiple versions. Take
the example of notebook-controller:

There are the following versions of manifests:

• The ones living in the upstream repo, used in development and testing.
• The base and overlays folders, in kubeflow/manifests. These were
  optimized to work with kfctl's kustomization combination magic.
• The base_v3 folder, designed to eschew vars and be buildable with
  kustomize. It imports files from base.

There is no need for such complexity and duplication. We will provide an example
(of the notebook-controller) and guide for how to consolidate them into a
proposed structure.

Deadlines

Date	Steps
2021/02/09	Steps 1-3

Common Questions

• Q: Will kfctl be supported?
  A: Kubeflow applications publish kustomize packages. Any tool that can
  work with kustomizations should work. Manifests are not tied to any specific
  deployment tool and more information on kfctl development can be found in
  the deployments-wg.

• Q: What Istio version will be included?
  A: Will be decided as part of step (3) of the plan above. In principle,
  you can use any version of your choosing, as long as it supports
  AuthorizationPolicy (1.4+).

cc @kubeflow/wg-manifests-leads