Can't start mimir

[ermiaqasemi]

I tried to deploy with the helm version on EKS but got the following error in all components, any idea?

ts=2023-03-09T17:39:03.467313448Z caller=mimir.go:802 level=error msg="module failed" module=memberlist-kv err="invalid service state: Stopping, expected: Running" ts=2023-03-09T17:39:03.467342253Z caller=mimir.go:802 level=error msg="module failed" module=store-gateway err="failed to start store-gateway, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running" ts=2023-03-09T17:39:03.467366388Z caller=mimir.go:802 level=error msg="module failed" module=runtime-config err="context canceled" ts=2023-03-09T17:39:03.467466312Z caller=server_service.go:55 level=info msg="server stopped" ts=2023-03-09T17:39:03.467500698Z caller=module_service.go:114 level=info msg="module stopped" module=server ts=2023-03-09T17:39:03.467555264Z caller=module_service.go:114 level=info msg="module stopped" module=sanity-check ts=2023-03-09T17:39:03.467589108Z caller=module_service.go:114 level=info msg="module stopped" module=usage-stats ts=2023-03-09T17:39:03.469599459Z caller=module_service.go:114 level=info msg="module stopped" module=activity-tracker ts=2023-03-09T17:39:03.469637238Z caller=mimir.go:791 level=info msg="Application stopped" ts=2023-03-09T17:39:03.469718257Z caller=log.go:65 level=error msg="error running application" err="failed services\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/__w/mimir/mimir/pkg/mimir/mimir.go:845\nmain.main\n\t/__w/mimir/mimir/cmd/mimir/main.go:212\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594" ts=2023-03-09T17:40:24.461952642Z caller=main.go:210 level=info msg="Starting application" version="(version=2.6.0, branch=HEAD, revision=27698f3)" ts=2023-03-09T17:40:24.46788123Z caller=server.go:323 level=info http=[::]:8080 grpc=[::]:9095 msg="server listening on addresses" ts=2023-03-09T17:40:24.474780847Z caller=memberlist_client.go:437 level=info msg="Using memberlist cluster label and node name" cluster_label= node=grafana-mimir-store-gateway-zone-b-0-1a6b47c2 ts=2023-03-09T17:40:24.476054246Z caller=inmemory.go:173 level=info msg="created in-memory index cache" maxItemSizeBytes=134217728 maxSizeBytes=1073741824 maxItems=maxInt ts=2023-03-09T17:40:24.476862394Z caller=module_service.go:82 level=info msg=initialising module=activity-tracker ts=2023-03-09T17:40:24.476922458Z caller=module_service.go:82 level=info msg=initialising module=sanity-check ts=2023-03-09T17:40:24.476994274Z caller=sanity_check.go:32 level=info msg="Checking directories read/write access" ts=2023-03-09T17:40:24.477098293Z caller=module_service.go:82 level=info msg=initialising module=usage-stats ts=2023-03-09T17:40:24.477161325Z caller=sanity_check.go:37 level=info msg="Directories read/write access successfully checked" ts=2023-03-09T17:40:24.477184884Z caller=sanity_check.go:39 level=info msg="Checking object storage config" ts=2023-03-09T17:40:24.484375195Z caller=sanity_check.go:44 level=info msg="Object storage config successfully checked" ts=2023-03-09T17:40:24.484446864Z caller=module_service.go:82 level=info msg=initialising module=server ts=2023-03-09T17:40:24.484542076Z caller=module_service.go:82 level=info msg=initialising module=runtime-config ts=2023-03-09T17:40:24.484785574Z caller=module_service.go:82 level=info msg=initialising module=memberlist-kv ts=2023-03-09T17:40:24.484908484Z caller=mimir.go:802 level=error msg="module failed" module=memberlist-kv err="invalid service state: Failed, expected: Running, failure: service &{0xc000b6c320 { true 10000000000 4 30000000000 200000000 3 30000000000 0 false 7946 false [dns+grafana-mimir-gossip-ring.monitoring.svc.cluster.local:7946] 1000000000 60000000000 10 false 0 300000000000 20000000000 0 {[] 7946 2000000000 5000000000 false 0xc000050aa0 false { false }} 0xc000050aa0 [{ringDesc 0xd48e20}]} 0xc000739b00 0xc000050aa0 0xc0006edb90 {[] {[] 0}} <nil> <nil> {0 0} map[] map[ringDesc:{ringDesc 0xd48e20}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc000673080 0xc000b68f60 0xc000b69020 0xc000b690e0 0xc000b691a0 0xc000b693e0 0xc000b69260 0xc000b694a0 0xc000b69320 0xc000b6adc0 0xc000b6ae00 0xc000b69620 0xc000b696e0 0xc000b69860 0xc000b697a0 0xc000306430 0xc0005dd340 0xc000306420 0xc000306428 0xc000b6ae40 0xc000b6ae80 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided" ts=2023-03-09T17:40:24.48499098Z caller=mimir.go:802 level=error msg="module failed" module=store-gateway err="failed to start store-gateway, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc000b6c320 { true 10000000000 4 30000000000 200000000 3 30000000000 0 false 7946 false [dns+grafana-mimir-gossip-ring.monitoring.svc.cluster.local:7946] 1000000000 60000000000 10 false 0 300000000000 20000000000 0 {[] 7946 2000000000 5000000000 false 0xc000050aa0 false { false }} 0xc000050aa0 [{ringDesc 0xd48e20}]} 0xc000739b00 0xc000050aa0 0xc0006edb90 {[] {[] 0}} <nil> <nil> {0 0} map[] map[ringDesc:{ringDesc 0xd48e20}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc000673080 0xc000b68f60 0xc000b69020 0xc000b690e0 0xc000b691a0 0xc000b693e0 0xc000b69260 0xc000b694a0 0xc000b69320 0xc000b6adc0 0xc000b6ae00 0xc000b69620 0xc000b696e0 0xc000b69860 0xc000b697a0 0xc000306430 0xc0005dd340 0xc000306420 0xc000306428 0xc000b6ae40 0xc000b6ae80 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided" ts=2023-03-09T17:40:24.485043163Z caller=module_service.go:114 level=info msg="module stopped" module=runtime-config ts=2023-03-09T17:40:24.485279556Z caller=server_service.go:55 level=info msg="server stopped" ts=2023-03-09T17:40:24.485309543Z caller=module_service.go:114 level=info msg="module stopped" module=server ts=2023-03-09T17:40:24.485360694Z caller=module_service.go:114 level=info msg="module stopped" module=usage-stats ts=2023-03-09T17:40:24.485406722Z caller=module_service.go:114 level=info msg="module stopped" module=sanity-check ts=2023-03-09T17:40:24.488448428Z caller=module_service.go:114 level=info msg="module stopped" module=activity-tracker ts=2023-03-09T17:40:24.488498555Z caller=mimir.go:791 level=info msg="Application stopped" ts=2023-03-09T17:40:24.488572886Z caller=log.go:65 level=error msg="error running application" err="failed services\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/__w/mimir/mimir/pkg/mimir/mimir.go:845\nmain.main\n\t/__w/mimir/mimir/cmd/mimir/main.go:212\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594

[ermiaqasemi]

Never mind, fine the issue :))

[pstibrany]

Can you share what the issue was and how you fixed it? It may help someone in the future who runs into this.

Main error message in your log is:

failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided