Bug: After configuration of grpc_tls_config and http_tls_config connection not possible

[GunFood]

What is the bug?

We configured Grafana Mimir to open port 9009 on tls only.
After a restart of Mimir a connection from Grafana (with http changed to https) doesn't work.

Mimir Log:

Mar 18 10:59:55 sys1234 systemd[1]: Started Horizontally scalable, highly available, multi-tenant, long term Prometheus..
Mar 18 10:59:55 sys1234 mimir[880096]: ts=2025-03-18T09:59:55.489535413Z caller=ruler.go:466 level=info msg="ruler is ACTIVE in the ring"
Mar 18 10:59:55 sys1234 mimir[880096]: ts=2025-03-18T09:59:55.489556122Z caller=ruler.go:471 level=info msg="ruler is only now starting to evaluate rules"
Mar 18 10:59:55 sys1234 mimir[880096]: ts=2025-03-18T09:59:55.489564348Z caller=ruler.go:525 level=info msg="ruler up and running"
Mar 18 10:59:55 sys1234 mimir[880096]: ts=2025-03-18T09:59:55.489586083Z caller=mimir.go:897 level=info msg="Application started"
Mar 18 10:59:55 sys1234 mimir[880096]: ts=2025-03-18T09:59:55.582123258Z caller=frontend_processor.go:84 level=error msg="error contacting frontend" address=127.0.0.1:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: read tcp 127.0.0.1:35214->127.0.0.1:9095: read: connection reset by peer\""
Mar 18 10:59:55 sys1234 mimir[880096]: ts=2025-03-18T09:59:55.623703913Z caller=frontend_processor.go:84 level=error msg="error contacting frontend" address=127.0.0.1:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: read tcp 127.0.0.1:35214->127.0.0.1:9095: read: connection reset by peer\""
Mar 18 10:59:55 sys1234 mimir[880096]: ts=2025-03-18T09:59:55.652981249Z caller=frontend_processor.go:84 level=error msg="error contacting frontend" address=127.0.0.1:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: read tcp 127.0.0.1:35214->127.0.0.1:9095: read: connection reset by peer\""

These last log entries are flooded the log.

Open Ports:

mimir 880096 mimir 3u IPv4 2425208 0t0 UDP localhost:48051->localhost:6831
mimir 880096 mimir 8u IPv6 2425217 0t0 TCP *:9009 (LISTEN)
mimir 880096 mimir 9u IPv6 2425218 0t0 TCP *:9095 (LISTEN)
mimir 880096 mimir 10u IPv6 2423736 0t0 TCP *:7946 (LISTEN)

After disabling tls config in Mimir, connection is fine and working again.

How to reproduce it?

Mimir Config:

multitenancy_enabled: false

blocks_storage:
backend: filesystem
bucket_store:
sync_dir: /tmp/mimir/tsdb-sync
filesystem:
dir: /opt/Mimir/data/tsdb
tsdb:
dir: /opt/Mimir/tsdb

compactor:
data_dir: /opt/Mimir/compactor
sharding_ring:
kvstore:
store: memberlist

distributor:
ring:
instance_addr: 127.0.0.1
kvstore:
store: memberlist

ingester:
ring:
instance_addr: 127.0.0.1
kvstore:
store: memberlist
replication_factor: 1

ruler_storage:
backend: filesystem
filesystem:
dir: /opt/Mimir/rules

server:
http_listen_port: 9009
log_level: warn
http_tls_config:
cert_file: /etc/mimir/237_grafana_test.pem
key_file: /etc/mimir/grafana_test.key
grpc_tls_config:
cert_file: /etc/mimir/237_grafana_test.pem
key_file: /etc/mimir/grafana_test.key

store_gateway:
sharding_ring:
replication_factor: 1

frontend:
log_queries_longer_than: 1s

limits:
compactor_blocks_retention_period: 1h

What did you think would happen?

Open tls connection on 9009

What was your environment?

We run Mimir version 2.15.0 on Redhat 9

Any additional context to share?

No response

[aknuds1]

Converting this to a discussion, as it looks more like a TLS (query-frontend) configuration problem than a bug, until more evidence is provided at least.

[GunFood]

I did a few more tests now and the issue occurs, then enabling grpc_tls_config. http_tls_config alone is working fine.