[Discussion] Do we deprecate or improve the `prune: false` template annotation injection?

[foot]

TL;DR

Is automatic injection of the kustomize.toolkit.fluxcd.io/prune: disabled annotation in CAPITemplates either

• useful and we should improve it, or is it
• dangerous (in that one could accidentally prevent deletion of expensive AWS resources) and we should remove the feature and ask template authors to manually add it where appropriate instead?

Background

When rendering a CAPITemplate we automatically inject a kustomize.toolkit.fluxcd.io/prune: disabled annotation into every resource in the spec.resourcetemplates that is not a cluster.x-k8s.io.Cluster and not a gitops.weave.works.GitopsCluster.

The intention here is stop flux from explicitly deleting subresources of the Cluster like AWSCluster, KubeadmControlPlane, AWSMachineTemplate etc and let the capi-controllers remove them itself.

This is the pattern recommended in the capi-quickstart guide https://cluster-api.sigs.k8s.io/user/quick-start.html#clean-up it has a nice big warning on that page that looks like this:

---

---

Current state of things

As of #1830 you can enabled / disable this automatic injection of prune: disabled annotations with an annotation on the template itself (templates.weave.works/inject-prune-annotation": "true")

• CAPITemplate: the default value for this annotation is "true" and we will inject the prune: false annotations
• GitopsTemplate: the default value for this annotation is "false" and we don't inject

This issue

If a kustomization / helmrelease is included in a CAPITemplate it receives the prune: false and is not cleaned up, this feels like a bug.

When considering how to fix this bug the question was posed whether we should keep this feature at all.

Pros:

• Nicer UX in that a template author doesn't have to understand the subtleties of CAPI x flux when authoring templates.

Cons:

• We can't really know exactly which resources to annotate with prune: false
  • Could accidentally leave expensive resources around
  • We can make a better guess than currently, e.g. prune: false on anything that is apiVersion: /.*cluster.x-k8s.io.*/ and not kind: Cluster.

[foot]

@LutzLange, @steve-fraser @darrylweaver it would be great to hear your thoughts having used the templates in anger a bit.

cc @davidstauffer @bigkevmcd

[foot]

There might be other solutions to the problem of proper cleanup of CAPI clusters, but haven't found anything yet

• Argo has a similar prune: false anno
• A recent writeup of argo + CAPI asks you to disconnect the cluster "app" and then manually kubectl delete cluster capi-eks, bit awkward.

[foot]

If we were to remove the feature, the "authoring a template" guide would gain a more in-depth section talking about capi + flux + prune behaviour.

[darrylweaver]

Just reviewing any outstanding issues and I came across this one.
The prune: false annotation should apply to CAPI resources to allow the CAPI controllers to manage and not flux.

Consider that hardly anyone will want to be burdened with this implementation detail unless they are CAPI devs.
This is something that the tooling should deal with for you.

Therefore please consider implementing a more intelligent approach of checking the apiversion and kind for CAPI resources and applying the prune annotation automatically to those resources, but not applying the annotation to other resource kinds.

[steve-fraser]

I found that doing it by hand (myself) determining the prune condition is the most valuable here and not automatically.

For example:

• Bucket creation for metrics for Thanos + Prometheus you want the bucket to be long lived past the cluster being destroyed, so you should add the annotation to that.
• CAPI resources you need juggle when for example you are changing instance types a new AWSMachineTemplate is created, but need the old AWSMachineTemplate to exist for CAPI to complete the transition.

As long as I can manually include the prune annotations that's what I will use.

Personally, I think you can remove the automatic injection functionality and leave it up to the customer