🐛 Deletion Deadlock

[spjmurray]

/kind bug

What steps did you take and what happened:

@mdbooth your input is treasured as per usual...

When I delete a cluster (properly this time, you can prevent Argo from deleting anything and let CAPI manage the whole process), what I'm seeing is control plane machines hanging.

From what I can tell there is a race where:

• CAPI adds a owner reference on the infrastructure
• I delete the Cluster
• CAPI should delete... the MDs, the KCP, then the infrastructure
• However the owner reference trigger the infrastructure delete early...

What happens is CAPO keeps trying to delete the infra and the CP machines at the same time.
As soon as the ports are detached from the network, that completes and the infrastructure gets deleted.
However the CP machines haven't been fully deleted yet, and cannot be deleted because they need to see the infrastructure resource in order to determine whether there's a loadbalancer that needs reconciling.

I expect either:

• CAPI needs to stop messing with owner references and let its internal ordering take precedence
• CAPO needs to cache infrastructure configuration in the machines so it doesn't need to refer to the OSC resource

What did you expect to happen:

You can actually delete things without a hang.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Cluster API Provider OpenStack version (Or git rev-parse HEAD if manually built): 0.7.1
• Cluster-API version: 1.3.2
• OpenStack version: Zed
• Minikube/KIND version:
• Kubernetes version (use kubectl version): 1.27
• OS (e.g. from /etc/os-release):

[tobiasgiese]

@spjmurray one known issue is that the cloud-config secret will be deleted. In this case CAPO cannot delete OpenStack resources as the requests to the API are no longer authenticated.
Maybe this is the issue in your case, as ArgoCD removes all resources.

[spjmurray]

Oh that was the first issue I ran into ages ago. Putting that in sync-wave -1 solves that issue... somehow. Apparently that's broken too, so most resource I create with argocd.argoproj.io/sync-options: Delete=false and let CAPI sort it out. That's actually discussed here: kubernetes-sigs/cluster-api#7624

[spjmurray]

Seems there is a related issue in CAPI proper kubernetes-sigs/cluster-api#7559.

[jichenjc]

CAPO needs to cache infrastructure configuration in the machines so it doesn't need to refer to the OSC resource

cache may introduce the inconsistency, I think solve this from logic instead of cache seems more reasonable

What happens is CAPO keeps trying to delete the infra and the CP machines at the same time.
As soon as the ports are detached from the network, that completes and the infrastructure gets deleted.
However the CP machines haven't been fully deleted yet, and cannot be deleted because they need to see the infrastructure resource in order to determine whether there's a loadbalancer that needs reconciling.

so can we delete the workload first then the infrastructure ? e.g through code logic A=> B => C
or something as you pointed out , use finalizer?

[spjmurray]

I guess it should be possible for the OSC manager to wait for all OSMs to disappear, let me give it a go...

[spjmurray]

Ah actually, I don't think that will work

{"ts":1685623244027.2656,"caller":"controller/controller.go:326","msg":"Reconciler error","controller":"openstackmachine","controllerGroup":"infrastructure.cluster.x-k8s.io","controllerKind":"OpenStackMachine","OpenStackMachine":{"name":"dynamic-control-plane-1c3990b5-npg2n","namespace":"dynamic"},"namespace":"dynamic","name":"dynamic-control-plane-1c3990b5-npg2n","reconcileID":"2628adb2-e431-476a-a98b-7dc961597608","err":"Resource not found: [GET https://nl1.eschercloud.com:9876/v2.0/lbaas/loadbalancers/f5a39020-abb1-4940-97f2-a586852a9af1], error message: {\"faultcode\": \"Client\", \"faultstring\": \"Load Balancer f5a39020-abb1-4940-97f2-a586852a9af1 not found.\", \"debuginfo\": null}"}
{"ts":1685623244027.3716,"caller":"controller/controller.go:326","msg":"Reconciler error","controller":"openstackmachine","controllerGroup":"infrastructure.cluster.x-k8s.io","controllerKind":"OpenStackMachine","OpenStackMachine":{"name":"dynamic-control-plane-1c3990b5-npg2n","namespace":"dynamic"},"namespace":"dynamic","name":"dynamic-control-plane-1c3990b5-npg2n","reconcileID":"cc634056-633c-4637-8fcf-4faf15f84897","err":"error getting infra provider cluster"}

It'll just moan constantly about the LB not existing. Ah right, wait until the machines vanish before deleting the LB perhaps. Hold my beer!

[spjmurray]

Codez are up and ready for critique... and it appears to work beautifully @jichenjc