What to do about dynamic creation of admission hooks and other resources?

[jlewi]

Misconfigured or misbehaving webhooks are an ongoing source of problems see:

• Admission Webhook prevents Kubeflow pod deployment. kubeflow#4808
• Uber Issue: KFServing admission hook causing widespread issues because its a global admission hook kserve/kserve#568
• Katib webhooks blocking pod creation in kubeflow namespace katib#1261

In #1213 we started adding testing to verify that webhooks are well behaved.

However, this only works if an application configures webhooks explicitly as part of its kustomize package.

We have at least two applications (katib, and kfserving) in which the webhooks are dynamically and automatically created. This has a couple disadvantages

1. Operator doesn't know beforehand what resources will be created
2. Operator has no way to customize the webhooks if needed
3. Controllers need sufficient RBAC permissions to create webhooks

Should we recommend or require that all applications support at least optionally explicit creation of their webhooks and provide YAML resources to do so?

Should we enforce that by adding tests similar to #1213 to disallow RBAC permissions to create webhooks?

/cc @andreyvelich @johnugeorge @ellistarn @yuzisun @cliveseldon @animeshsingh

[issue-label-bot]

Issue Label Bot is not confident enough to auto-label this issue.
See dashboard for more details.

[yuzisun]

@jlewi As of KFServing 0.3 which is in kubeflow 1.1 release branch now the webhook is no longer auto-generated, see here.

[andreyvelich]

@jlewi I think we can add Katib YAML webhooks as part of Kustomize package. That can also help when we clean-up resources after Kubeflow deletion.
What do you think @gaocegege @johnugeorge ?

[ellistarn]

For a little bit of insight on this, Kubebuilder 1.0 used dynamic webhooks, so any controller built on that version will have this (Katib, etc). Kubebuilder 2.0 uses static webhooks.

[jlewi]

Thanks @ellistarn for the explanation. So it looks like we should probably figure out which controllers if any besides katib are still using dynamic webhooks and get them updated.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in one week if no further activity occurs. Thank you for your contributions.

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
area/engprod	0.88
area/kfctl	0.55

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.