Istio with mimir have rpc errors in querier

[kssrao87]

Hi team,
Configured mimir on kubernetes with backend to s3 by changing values.yaml file under https://github.com/grafana/mimir/blob/main/operations/helm/charts/mimir-distributed/values.yaml

I have seen errors in querier logs,

ts=2023-04-12T08:12:21.626764494Z caller=mimir.go:789 level=info msg="Application started"
ts=2023-04-12T08:12:21.636237884Z caller=worker.go:232 level=info msg="adding connection" addr=172.30.116.217:9095 in-use=true
2023/04/12 08:12:21 INFO: [core] [Channel #3] Channel created
2023/04/12 08:12:21 INFO: [core] [Channel #3] original dial target is: "172.30.116.217:9095"
2023/04/12 08:12:21 INFO: [core] [Channel #3] dial target "172.30.116.217:9095" parse failed: parse "172.30.116.217:9095": first path segment in URL cannot contain colon
2023/04/12 08:12:21 INFO: [core] [Channel #3] fallback to scheme "passthrough"
2023/04/12 08:12:21 INFO: [core] [Channel #3] parsed dial target is: {Scheme:passthrough Authority: Endpoint:172.30.116.217:9095 URL:{Scheme:passthrough Opaque: User: Host: Path:/172.30.116.217:9095 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2023/04/12 08:12:21 INFO: [core] [Channel #3] Channel authority set to "172.30.116.217:9095"
2023/04/12 08:12:21 INFO: [core] [Channel #3] Resolver state updated: {
  "Addresses": [
    {
      "Addr": "172.30.116.217:9095",
      "ServerName": "",
      "Attributes": null,
      "BalancerAttributes": null,
      "Type": 0,
      "Metadata": null
    }
  ],
  "ServiceConfig": null,
  "Attributes": null
} (resolver returned new addresses)
2023/04/12 08:12:21 INFO: [core] [Channel #3] Channel switches to new LB policy "pick_first"
2023/04/12 08:12:21 INFO: [core] [Channel #3 SubChannel #4] Subchannel created
2023/04/12 08:12:21 INFO: [core] blockingPicker: the picked transport is not ready, loop back to repick
2023/04/12 08:12:21 INFO: [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to CONNECTING
2023/04/12 08:12:21 INFO: [core] [Channel #3 SubChannel #4] Subchannel picks a new address "172.30.116.217:9095" to connect
2023/04/12 08:12:21 INFO: [core] blockingPicker: the picked transport is not ready, loop back to repick
2023/04/12 08:12:21 INFO: [core] blockingPicker: the picked transport is not ready, loop back to repick
2023/04/12 08:12:21 INFO: [core] blockingPicker: the picked transport is not ready, loop back to repick
2023/04/12 08:12:21 INFO: [core] blockingPicker: the picked transport is not ready, loop back to repick
2023/04/12 08:12:21 INFO: [core] blockingPicker: the picked transport is not ready, loop back to repick
2023/04/12 08:12:21 INFO: [core] blockingPicker: the picked transport is not ready, loop back to repick
2023/04/12 08:12:21 INFO: [core] blockingPicker: the picked transport is not ready, loop back to repick
2023/04/12 08:12:21 INFO: [core] blockingPicker: the picked transport is not ready, loop back to repick
2023/04/12 08:12:21 INFO: [core] blockingPicker: the picked transport is not ready, loop back to repick
2023/04/12 08:12:21 INFO: [core] blockingPicker: the picked transport is not ready, loop back to repick
2023/04/12 08:12:21 INFO: [core] blockingPicker: the picked transport is not ready, loop back to repick
2023/04/12 08:12:21 INFO: [core] blockingPicker: the picked transport is not ready, loop back to repick
2023/04/12 08:12:21 INFO: [core] blockingPicker: the picked transport is not ready, loop back to repick
2023/04/12 08:12:21 INFO: [core] blockingPicker: the picked transport is not ready, loop back to repick
2023/04/12 08:12:21 INFO: [core] blockingPicker: the picked transport is not ready, loop back to repick
2023/04/12 08:12:21 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0xc000f8f4a0, {CONNECTING <nil>}
2023/04/12 08:12:21 INFO: [core] [Channel #3] Channel Connectivity change to CONNECTING
2023/04/12 08:12:21 INFO: [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to READY
2023/04/12 08:12:21 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0xc000f8f4a0, {READY <nil>}
2023/04/12 08:12:21 INFO: [core] [Channel #3] Channel Connectivity change to READY
ts=2023-04-12T08:12:21.672865004Z caller=scheduler_processor.go:123 level=error msg="error processing requests from scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.217:9095
ts=2023-04-12T08:12:21.672888124Z caller=scheduler_processor.go:123 level=error msg="error processing requests from scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.217:9095
ts=2023-04-12T08:12:21.672961478Z caller=scheduler_processor.go:123 level=error msg="error processing requests from scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.217:9095
ts=2023-04-12T08:12:21.673021132Z caller=scheduler_processor.go:123 level=error msg="error processing requests from scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.217:9095
ts=2023-04-12T08:12:21.673039855Z caller=scheduler_processor.go:123 level=error msg="error processing requests from scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.217:9095
ts=2023-04-12T08:12:21.673079236Z caller=scheduler_processor.go:123 level=error msg="error processing requests from scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.217:9095
ts=2023-04-12T08:12:21.673096639Z caller=scheduler_processor.go:123 level=error msg="error processing requests from scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.217:9095
ts=2023-04-12T08:12:21.673134133Z caller=scheduler_processor.go:123 level=error msg="error processing requests from scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.217:9095
ts=2023-04-12T08:12:21.67314837Z caller=scheduler_processor.go:123 level=error msg="error processing requests from scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.217:9095
ts=2023-04-12T08:12:21.673208992Z caller=scheduler_processor.go:123 level=error msg="error processing requests from scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.217:9095
ts=2023-04-12T08:12:21.673241748Z caller=scheduler_processor.go:123 level=error msg="error processing requests from scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.217:9095
ts=2023-04-12T08:12:21.673311678Z caller=scheduler_processor.go:123 level=error msg="error processing requests from scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.217:9095
ts=2023-04-12T08:12:21.673376155Z caller=scheduler_processor.go:123 level=error msg="error processing requests from scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.217:9095
ts=2023-04-12T08:12:21.673429951Z caller=scheduler_processor.go:123 level=error msg="error processing requests from scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.217:9095
ts=2023-04-12T08:12:21.673469488Z caller=scheduler_processor.go:123 level=error msg="error processing requests from scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.217:9095
ts=2023-04-12T08:12:21.673502959Z caller=scheduler_processor.go:123 level=error msg="error processing requests from scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.217:9095
ts=2023-04-12T08:12:21.926190172Z caller=scheduler_processor.go:123 level=error msg="error processing requests from scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.217:9095
ts=2023-04-12T08:12:21.965764053Z caller=scheduler_processor.go:123 level=error msg="error processing requests from scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.217:9095
ts=2023-04-12T08:12:21.971712361Z caller=scheduler_processor.go:123 level=error msg="error processing requests from scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.217:9095

And also in query-frontend

2023/04/12 11:27:29 INFO: [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to IDLE
2023/04/12 11:27:29 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0xc0005f27c0, {IDLE connection error: desc = "transport: Error while dialing dial tcp 172.30.116.215:9095: connect: connection refused"}
2023/04/12 11:27:29 INFO: [core] [Channel #3] Channel Connectivity change to IDLE
2023/04/12 11:27:29 INFO: [core] [Channel #5 SubChannel #6] Subchannel Connectivity change to CONNECTING
2023/04/12 11:27:29 INFO: [core] [Channel #5 SubChannel #6] Subchannel picks a new address "172.30.71.223:9095" to connect
2023/04/12 11:27:29 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0xc0004ce8f0, {CONNECTING <nil>}
2023/04/12 11:27:29 INFO: [core] [Channel #5] Channel Connectivity change to CONNECTING
2023/04/12 11:27:29 INFO: [core] [Channel #5 SubChannel #6] Subchannel Connectivity change to READY
2023/04/12 11:27:29 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0xc0004ce8f0, {READY <nil>}
2023/04/12 11:27:29 INFO: [core] [Channel #5] Channel Connectivity change to READY
2023/04/12 11:27:29 INFO: [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to CONNECTING
2023/04/12 11:27:29 INFO: [core] [Channel #3 SubChannel #4] Subchannel picks a new address "172.30.116.215:9095" to connect
2023/04/12 11:27:29 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0xc0005f27c0, {CONNECTING <nil>}
2023/04/12 11:27:29 INFO: [core] [Channel #3] Channel Connectivity change to CONNECTING
2023/04/12 11:27:29 INFO: [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to READY
2023/04/12 11:27:29 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0xc0005f27c0, {READY <nil>}
2023/04/12 11:27:29 INFO: [core] [Channel #3] Channel Connectivity change to READY
ts=2023-04-12T11:27:30.133919544Z caller=frontend_scheduler_worker.go:286 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.71.223:9095
ts=2023-04-12T11:27:30.153136755Z caller=frontend_scheduler_worker.go:286 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.71.223:9095
ts=2023-04-12T11:27:30.153345025Z caller=frontend_scheduler_worker.go:286 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.71.223:9095
ts=2023-04-12T11:27:30.153473359Z caller=frontend_scheduler_worker.go:286 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.71.223:9095
ts=2023-04-12T11:27:30.207264973Z caller=frontend_scheduler_worker.go:286 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.71.223:9095
ts=2023-04-12T11:27:30.520532286Z caller=frontend_scheduler_worker.go:286 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.215:9095
ts=2023-04-12T11:27:30.523368326Z caller=frontend_scheduler_worker.go:286 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.215:9095
ts=2023-04-12T11:27:30.523521714Z caller=frontend_scheduler_worker.go:286 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.215:9095
ts=2023-04-12T11:27:30.740116026Z caller=frontend_scheduler_worker.go:286 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.215:9095
ts=2023-04-12T11:27:30.936452576Z caller=frontend_scheduler_worker.go:286 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.215:9095
ts=2023-04-12T11:27:31.172651342Z caller=frontend_scheduler_worker.go:286 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.71.223:9095
ts=2023-04-12T11:27:31.19602701Z caller=frontend_scheduler_worker.go:286 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.71.223:9095
ts=2023-04-12T11:27:31.538030527Z caller=frontend_scheduler_worker.go:286 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.71.223:9095
ts=2023-04-12T11:27:31.556781546Z caller=frontend_scheduler_worker.go:286 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.71.223:9095
ts=2023-04-12T11:27:31.958722468Z caller=frontend_scheduler_worker.go:286 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.215:9095
ts=2023-04-12T11:27:32.032885791Z caller=frontend_scheduler_worker.go:286 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.71.223:9095
ts=2023-04-12T11:27:32.111935211Z caller=frontend_scheduler_worker.go:286 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.215:9095
ts=2023-04-12T11:27:32.279897609Z caller=frontend_scheduler_worker.go:286 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.215:9095
ts=2023-04-12T11:27:32.334525905Z caller=frontend_scheduler_worker.go:286 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.215:9095
ts=2023-04-12T11:27:32.788151443Z caller=frontend_scheduler_worker.go:286 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.215:9095
ts=2023-04-12T11:27:32.958997489Z caller=frontend_scheduler_worker.go:286 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.71.223:9095
ts=2023-04-12T11:27:32.962871629Z caller=frontend_scheduler_worker.go:286 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.71.223:9095
ts=2023-04-12T11:27:33.14432899Z caller=frontend_scheduler_worker.go:286 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.71.223:9095
ts=2023-04-12T11:27:33.320382272Z caller=frontend_scheduler_worker.go:286 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.215:9095
ts=2023-04-12T11:27:33.354728688Z caller=frontend_scheduler_worker.go:286 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.71.223:9095
ts=2023-04-12T11:27:33.56266284Z caller=frontend_scheduler_worker.go:286 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.71.223:9095
ts=2023-04-12T11:27:33.905581583Z caller=frontend_scheduler_worker.go:286 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.215:9095
ts=2023-04-12T11:27:33.91856001Z caller=frontend_scheduler_worker.go:286 level=error msg="error sending requests to scheduler" err=EOF addr=172.30.116.215:9095
ts=2023-04-12T11:27:34.059704031Z caller=frontend_scheduler_worker.go:286 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.116.215:9095
ts=2023-04-12T11:27:34.316369615Z caller=frontend_scheduler_worker.go:286 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.71.223:9095
ts=2023-04-12T11:27:34.448474017Z caller=frontend_scheduler_worker.go:286 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=172.30.71.223:9095

[kssrao87]

Based on the earlier discussion, is it recommended to use mimir without istio?

[alextricity25]

I'm not sure, but to me that sounds like there is no path forward to support istio with the mimir helm chart. I'm seeing the same issue that you are seeing. I'll post something down below.

[alextricity25]

Hi everyone 👋 I'm seeing the exact same thing.

I'm installing the mimir-distributed helm chart (chart version 5.3.0) on a k8s environment with istio configured in STRICT mode and using the REGISTRY_ONLY option. The helm chart installation goes through just fine, but when trying to access the /api/v1/status/buildinfo endpoint, I get a 502 error. The logs in the query-frontend pod have the following:

query-frontend ts=2024-05-16T20:05:20.67279404Z caller=frontend_scheduler_worker.go:325 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=10.44.3.64:9095
query-frontend ts=2024-05-16T20:05:21.015667935Z caller=frontend_scheduler_worker.go:325 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=10.44.3.64:9095
query-frontend ts=2024-05-16T20:05:21.029982125Z caller=frontend_scheduler_worker.go:325 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=10.44.4.42:9095
query-frontend ts=2024-05-16T20:05:21.059571974Z caller=frontend_scheduler_worker.go:325 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=10.44.4.42:9095
query-frontend ts=2024-05-16T20:05:21.33433157Z caller=frontend_scheduler_worker.go:325 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=10.44.3.64:9095
query-frontend ts=2024-05-16T20:05:21.386713219Z caller=frontend_scheduler_worker.go:325 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=10.44.4.42:9095
query-frontend ts=2024-05-16T20:05:21.501432587Z caller=frontend_scheduler_worker.go:325 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=10.44.4.42:9095
query-frontend ts=2024-05-16T20:05:21.628827346Z caller=frontend_scheduler_worker.go:325 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=10.44.3.64:9095
query-frontend ts=2024-05-16T20:05:21.774282664Z caller=frontend_scheduler_worker.go:325 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=10.44.3.64:9095
query-frontend ts=2024-05-16T20:05:22.03580881Z caller=frontend_scheduler_worker.go:325 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=10.44.4.42:9095
query-frontend ts=2024-05-16T20:05:22.04177938Z caller=frontend_scheduler_worker.go:325 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=10.44.3.64:9095
query-frontend ts=2024-05-16T20:05:22.122381949Z caller=frontend_scheduler_worker.go:325 level=error msg="error sending requests to scheduler" err="rpc error: code = Unavailable desc = " addr=10.44.4.42:9095

[mrehm96]

I have the very same problem. Were you able to fix it in the meantime?

[alextricity25]

No :/ I ended up dropping Istio from my telemetry cluster

[mrehm96]

I could finally fix it now by applying the fix proposed in https://github.com/owais/istio-grpc-headless-test/tree/main (changing the port names). I guess this is also related to this: https://grafana.com/docs/loki/latest/setup/install/istio/.

[sirishkumar]

What worked for me are following settings

1. Explicitly set appProtocol to 'tcp' for all grpc endpoints in service definitions. This results in creating listeners for endpoints defined in headless services.

 - appProtocol: tcp                                                                                                                                                                                                                                                                                                                   
     name: grpc                                                                                                                                                                                                                                                                                                                          
     port: 9095                                                                                                                                                                                                                                                                                                                          
     protocol: TCP                                                                                                                                                                                                                                                                                                                       
     targetPort: grpc  

2. Create headless service for query_frontend to allow POD to POD communication in istio using IP(Querier -> Frontend). Helm chart disables creating headless service for query frontend when scheduler is enabled.

[ttsang1801]

Hi @sirishkumar, I am facing a similar problem and noticed that you mentioned setting the appProtocol to tcp for all gRPC endpoints. However, I couldn't find where to set the appProtocol in the latest mimir-distributed chart. (https://github.com/grafana/mimir/blob/main/operations/helm/charts/mimir-distributed/values.yaml) Could you please guide me on that?

[sirishkumar]

@ttsang1801 Sorry for the delay, we use Pulumi for deployment and we set it using resource transformations https://www.pulumi.com/docs/iac/concepts/options/transforms/

[sang-truong-tan]

Unsure if I've missed something, but in my case, it requires me to add appProtocol: tcp to all ports for all service to enable the communication