Operator calling the wrong pod endpoint when trying to scale down ingesters

[grecuionut]

We are trying to scale down ingesters using the MutatingAdmissionWebhook as described here.

The required labels and annotations were added to the objects as shown below:

# labels
grafana.com/prepare-downscale=true

# annotations
grafana.com/prepare-downscale-http-path=ingester/prepare-shutdown
grafana.com/prepare-downscale-http-port=8080

When trying the scale down the statefulset/mimir-ingester-zone-a, the operator failing to resolve the pod when sending HTTP post request, as the fqdn is constructed as <pod_name>.<service_name>.<namespace>.svc.cluster.local.

mimir-ingester-zone-a-1.mimir-ingester-zone-a.mimir.svc.cluster.local

These are the existing services:

mimir-ingester-headless                  ClusterIP   None             <none>        8080/TCP,9095/TCP   19d
mimir-ingester-zone-a                    ClusterIP   <ip_address>     <none>        8080/TCP,9095/TCP   19d
mimir-ingester-zone-b                    ClusterIP   <ip_address>    <none>        8080/TCP,9095/TCP   19d
mimir-ingester-zone-c                    ClusterIP   <ip_address>    <none>        8080/TCP,9095/TCP   19d

In order to resolve the pod, the headless service should be used instead (mimir-ingester-headless). More info

Operator logs

level=error ts=2024-01-16T13:29:51.838387358Z name=mimir-ingester-zone-a resource=statefulsets namespace=mimir request_gvk="autoscaling/v1, Kind=Scale" old_replicas=2 new_replicas=1 url=mimir-ingester-zone-a-1.mimir-ingester-zone-a.mimir.svc.cluster.local:443/ingester/prepare-shutdown index=1 msg="error sending HTTP post request" err="Post \"http://mimir-ingester-zone-a-1.mimir-ingester-zone-a.mimir.svc.cluster.local:443/ingester/prepare-shutdown\": dial tcp: lookup mimir-ingester-zone-a-1.mimir-ingester-zone-a.mimir.svc.cluster.local on <name_server_ip>:53: no such host"

[jhychan]

If I'm not mistaken, the generated service name for a statefulset comes from the serviceName field in the StatefulSet.

Having looked over a number of the Loki/Mimir/Tempo helm charts we currently use, many of them do not correctly template the serviceName against the appropriate headless Service.

[stephcan]

@grecuionut did you use a workaround for this?

[PabloPie]

Hello @andyasp, has this issue been acknowledged? There is an associated PR (#173 ) to fix it but it hasn't been acknowledged for months.

[andyasp]

Hello!

I took a look and I think the confusion derives from a discrepancy between the Jsonnet and Helm chart of Mimir. There are tests in the Mimir repository that generate manifests for each that make it easier to see.

First, here's the definition of a headless service from the linked Kubernetes docs:

To define a headless Service, you make a Service with .spec.type set to ClusterIP (which is also the default for type), and you additionally set .spec.clusterIP to None.

In Mimir's Helm the ingester-zone-a service is not headless (reference). Instead, there is an additional headless ingester service (reference).

In Mimir's Jsonnet, ingester-zone-a service is headless (reference) and there is no additional headless service.

So the existing behavior works for Mimir's Jsonnet (and unspecified other users), but doesn't with Mimir's Helm.

With that in mind, #173 seems incorrect. Assuming there is a -headless service breaks existing users. Attempting to be compatible by trying both variations would also be less than ideal for both.

A way forward may be to expose a configurable option (not sure where would be best currently) and preserve the existing behavior as the default to not break current users.

Edit: I looked at this again and I think I was wrong and the proposed change may be correct in its approach after all. I said the above thinking the Helm's StatefulSet's serviceName pointed to the non-headless service name, but that is not the case (reference). The Jsonnet's StatefulSet serviceName also works (reference). This makes sense given the Kubernetes documentation about how that field is used here:

Each Pod in a StatefulSet derives its hostname from the name of the StatefulSet and the ordinal of the Pod. The pattern for the constructed hostname is $(statefulset name)-$(ordinal). The example above will create three Pods named web-0,web-1,web-2. A StatefulSet can use a Headless Service to control the domain of its Pods. The domain managed by this Service takes the form: $(service name).$(namespace).svc.cluster.local, where "cluster.local" is the cluster domain. As each Pod is created, it gets a matching DNS subdomain, taking the form: $(podname).$(governing service domain), where the governing service is defined by the serviceName field on the StatefulSet.

[PabloPie]

Thanks a lot for taking the time to look into this Andy! We are currently running a fork of the operator but would love to move back to using upstream. What do you propose as a way forward? Is the open PR good as it is or does it need additional changes?

I've seen there is some work being done to better integrate the rollout operator into the Mimir helm chart (grafana/mimir#6733) so I'm assuming more people are going to start facing these same issues.

[andyasp]

We are currently running a fork of the operator but would love to move back to using upstream.

Sorry for the frustration!

What do you propose as a way forward? Is the open PR good as it is or does it need additional changes?

I'll likely end up reimplementing a fix to prevent a back and forth from stalling this further. Nothing else is needed at this time and I'll try to resolve this issue before the next release.

I've seen there is some work being done to better integrate the rollout operator into the Mimir helm chart

There was some discussion recently here as well: grafana/helm-charts#3656.

[andyasp]

The fix has been merged in #221. That change will appear beginning with v0.27.0. My estimate of when that release will be cut would be the week of May 19th. FYI @PabloPie

[PabloPie]

Thanks a lot @andyasp, we'll give it a try once it's out.