Enhance Citus Runbook Scripts

[jnels124]

Problem

After recent updates to citus runbooks there are still some adjustments needed.

Solution

1. Adjust restore snapshot script to skip rotating to coord-0 and making sure replica is caught up
2. Ensure scripts are idempotent and able to be re-ran when there is a failure in the middle of the script on a previous run
3. Make master version upgrade optional in upgrade k8s script
4. increase shutdown timeout to 10 minutes and add retries
5. Detect pods stuck in scale down state. Verify annotation exists, then delete manually

Alternatives

No response

[jnels124]

address this failure as well

2025-07-25T19:15:32Z Waiting for 1 replicas in statefulset.apps/mirror-citus-shard2
statefulset.apps/mirror-citus-shard2 condition met
2025-07-25T19:15:37Z No failover needed: 'mirror-citus-coord-0' is already primary for group 0.
2025-07-25T19:15:38Z No failover needed: 'mirror-citus-shard0-0' is already primary for group 1.
2025-07-25T19:15:40Z No failover needed: 'mirror-citus-shard1-0' is already primary for group 2.
2025-07-25T19:15:41Z No failover needed: 'mirror-citus-shard2-0' is already primary for group 3.
2025-07-25T19:15:43Z Expected 4 master pods and found only 1 in namespace previewnet-citus