[epic] Ensure no two ClusterExtensions can manage the same underlying object

[joelanford]

For v1.0.0 it was decided that utilizing the underlying Helm management when concurrent reconciliations is set to a value of 1 is sufficient. While the underlying implementation details may change to prevent race conditions when scaling concurrent reconciliations > 1, the behavior that no two ClusterExtensions should be able to manage the same underlying objects needs to be documented.

Documentation should be contributed to the docs/drafts/ directory

Issues:

• Docs: Single ClusterExtension ownership concept #1247

[joelanford]

I spent a little time hacking on this in the Rukpak repo and came up with:

---
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingAdmissionPolicy
metadata:
  name: single-owner-policy
spec:
  failurePolicy: Fail
  matchConstraints:
    objectSelector:
      matchExpressions:
      - key: "core.rukpak.io/owner-name"
        operator: Exists
    resourceRules:
    - apiGroups:
      - "*"
      apiVersions:
      - "*"
      resources:
      - "*"
      operations:
      - UPDATE

  validations:
  - expression: |
      !("core.rukpak.io/owner-name" in oldObject.metadata.labels) ||
      !("core.rukpak.io/owner-name" in object.metadata.labels) ||
      oldObject.metadata.labels["core.rukpak.io/owner-name"] == object.metadata.labels["core.rukpak.io/owner-name"]
    messageExpression: '"object is owned by BundleDeployment \"" + oldObject.metadata.labels["core.rukpak.io/owner-name"]
      + "\": cannot be adopted by BundleDeployment \"" + object.metadata.labels["core.rukpak.io/owner-name"]
      + "\""'
    reason: Invalid
---
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: single-owner-policy
spec:
  policyName: single-owner-policy
  validationActions:
  - Deny

It will need to be adapted to change "core.rukpak.io/owner-name" to whatever label we decide to inject from the ClusterExtension API, but other than that my manual tests seemed to do the right thing.

I noticed that testing this in "real" scenarios may be difficult because Helm has its own logic to implement a single-owner policy. Perhaps we could write a test that tries to make concurrent changes to see if we can make if flaky with respect to the race conditions we expect, and then add this policy and check that the flakes disappear.

In addition, I think we can also just have some direct envtest-based tests that go around helm and try to make label changes directly so that we mimic helm but avoid its single-owner logic. That way, we can implement assertion logic based on what we expect this policy to do.