Rollout operator does not respect `grafana.com/min-time-between-zones-downscale` label

[verejoel]

Perhaps I have misunderstood the meaning of this label. My expectation is that if I set it to 5m, the rollout operator will wait at least 5 minutes from the point that the previous stateful set in the rollout group is ready, to begin rolling out the next statefulset.

I have three statefulsets defined, with the following labels/annotations:

# Zone A
  annotations:
    rollout-max-unavailable: "2"
  labels:
    controller.receive.thanos.io: thanos-hashring-controller
    controller.receive.thanos.io/hashring: soft-tenants
    grafana.com/min-time-between-zones-downscale: 5m
    name: thanos-ingester-zone-a
    rollout-group: ingester

# Zone B
  annotations:
    grafana.com/rollout-downscale-leader: thanos-ingester-zone-a
    rollout-max-unavailable: "2"
  labels:
    controller.receive.thanos.io: thanos-hashring-controller
    controller.receive.thanos.io/hashring: soft-tenants
    grafana.com/min-time-between-zones-downscale: 5m
    name: thanos-ingester-zone-b
    rollout-group: ingester

# Zone C
  annotations:
    grafana.com/rollout-downscale-leader: thanos-ingester-zone-b
    rollout-max-unavailable: "2"
  labels:
    controller.receive.thanos.io: thanos-hashring-controller
    controller.receive.thanos.io/hashring: soft-tenants
    grafana.com/min-time-between-zones-downscale: 5m
    name: thanos-ingester-zone-c
    rollout-group: ingester

However, when I trigger a rollout with this config, each zone is rolled out immediately once the leader is ready:

❯ kt rollout restart sts --selector=app.kubernetes.io/component=ingester

# 47 seconds later - zone A is fully ready, zone B is immediately torn down
❯ kt get pods | grep ingester
thanos-ingester-zone-a-0                      1/1     Running       0          47s
thanos-ingester-zone-a-1                      1/1     Running       0          47s
thanos-ingester-zone-b-0                      1/1     Terminating   0          11m
thanos-ingester-zone-b-1                      1/1     Terminating   0          11m
thanos-ingester-zone-c-0                      1/1     Running       0          11m
thanos-ingester-zone-c-1                      1/1     Running       0          11m

# same for zone C
❯ kt get pods | grep ingester
thanos-ingester-zone-a-0                      1/1     Running       0          81s
thanos-ingester-zone-a-1                      1/1     Running       0          81s
thanos-ingester-zone-b-0                      1/1     Running       0          26s
thanos-ingester-zone-b-1                      1/1     Running       0          26s
thanos-ingester-zone-c-0                      0/1     Running       0          1s
thanos-ingester-zone-c-1                      1/1     Terminating   0          11m

Is this a bug? A wrong expectation? Or am I misconfiguring something?

[colega]

the rollout operator will wait at least 5 minutes from the point that the previous stateful set in the rollout group is ready, to begin rolling out the next statefulset.

I think you didn't understand it right. min-time-between-zones-downscale is used in conjunction with grafana.com/rollout-downscale-leader label and it coordinates the replicas changes in the follower ensuring that a minimum amount of time has passed since the leader was scaled down.

It has nothing to do with restarts or rollouts.

[verejoel]

Understood! Do you think there would be scope to add-in this functionality?

My use-case is to allow an AZ to stabilize between rollouts, similar to minReadySeconds would allow me to let individual pods to stabilize on a rollout.

[colega]

Maybe this is something we can achieve using standard Kubernetes readinessProbe config? Combining periodSeconds & successThreshold?

[verejoel]

We need the functionality of minReadySeconds. For example, pod IP addresses aren't published until the pod is Ready, and in our case, we need to be sure that the new DNS entries have propagated to clients. Thus the usefulness of minReadySeconds.

How about expanding the operator to respect minReadySeconds if set on the stateful set?