store-gateway fails to start

[stehnes]

Hi,

I try to install Grafana Mimir in our test environment, but I cannot succeed with the store-gateway. It refuses to start.
I used the Helm chart mimir-distributed

level=info ts=2022-11-07T17:05:34.77816406Z caller=main.go:210 msg="Starting application" version="(version=2.4.0, branch=HEAD, revision=32137ee)"
level=info ts=2022-11-07T17:05:34.781553301Z caller=server.go:306 http=[::]:8080 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2022-11-07T17:05:34.787351735Z caller=memberlist_client.go:436 msg="Using memberlist cluster label and node name" cluster_label= node=mimir-store-gateway-0-e1d30a8c
level=info ts=2022-11-07T17:05:34.880167491Z caller=memcached.go:49 msg="created memcached cache"
level=info ts=2022-11-07T17:05:34.884363628Z caller=memberlist_client.go:543 msg="memberlist fast-join starting" nodes_found=11 to_join=8
level=info ts=2022-11-07T17:05:34.884850153Z caller=memcached.go:49 msg="created memcached cache"
level=info ts=2022-11-07T17:05:34.890042302Z caller=memcached.go:60 msg="created memcached index cache"
level=info ts=2022-11-07T17:05:34.891116198Z caller=module_service.go:82 msg=initialising module=sanity-check
level=info ts=2022-11-07T17:05:34.89138058Z caller=sanity_check.go:32 msg="Checking directories read/write access"
level=info ts=2022-11-07T17:05:34.891608112Z caller=sanity_check.go:37 msg="Directories read/write access successfully checked"
level=info ts=2022-11-07T17:05:34.891659162Z caller=sanity_check.go:39 msg="Checking object storage config"
level=info ts=2022-11-07T17:05:34.891953161Z caller=module_service.go:82 msg=initialising module=activity-tracker
level=info ts=2022-11-07T17:05:34.891210375Z caller=module_service.go:82 msg=initialising module=usage-stats
level=info ts=2022-11-07T17:05:34.978116033Z caller=memberlist_client.go:563 msg="memberlist fast-join finished" joined_nodes=8 elapsed_time=100.300178ms
level=info ts=2022-11-07T17:05:34.978294975Z caller=memberlist_client.go:576 msg="joining memberlist cluster" join_members=dns+mimir-gossip-ring.mimir.svc.cluster.local:7946
level=info ts=2022-11-07T17:05:35.185313291Z caller=memberlist_client.go:595 msg="joining memberlist cluster succeeded" reached_nodes=9 elapsed_time=207.019919ms
level=info ts=2022-11-07T17:05:35.250803413Z caller=reporter.go:110 msg="usage stats reporter initialized" cluster_id=37ac06c4-473d-4587-b1e9-ed370f4b08c7
level=info ts=2022-11-07T17:05:35.252927055Z caller=sanity_check.go:44 msg="Object storage config successfully checked"
level=info ts=2022-11-07T17:05:35.253096315Z caller=module_service.go:82 msg=initialising module=server
level=info ts=2022-11-07T17:05:35.253190035Z caller=module_service.go:82 msg=initialising module=runtime-config
level=info ts=2022-11-07T17:05:35.253492161Z caller=module_service.go:82 msg=initialising module=memberlist-kv
level=info ts=2022-11-07T17:05:35.253556935Z caller=module_service.go:82 msg=initialising module=store-gateway
level=info ts=2022-11-07T17:05:35.253838782Z caller=basic_lifecycler.go:261 msg="instance not found in the ring" instance=mimir-store-gateway-0 ring=store-gateway
level=info ts=2022-11-07T17:05:35.254801874Z caller=gateway.go:201 msg="waiting until store-gateway is JOINING in the ring"
level=info ts=2022-11-07T17:05:35.254834917Z caller=gateway.go:205 msg="store-gateway is JOINING in the ring"
level=info ts=2022-11-07T17:05:35.25484844Z caller=bucket_stores.go:165 msg="synchronizing TSDB blocks for all users"
level=warn ts=2022-11-07T17:05:40.325435736Z caller=bucket_stores.go:170 msg="failed to synchronize TSDB blocks" err="The specified key does not exist."
level=info ts=2022-11-07T17:05:40.325592236Z caller=basic_lifecycler.go:202 msg="ring lifecycler is shutting down" ring=store-gateway
level=info ts=2022-11-07T17:05:40.325878848Z caller=basic_lifecycler.go:372 msg="unregistering instance from ring" ring=store-gateway
level=info ts=2022-11-07T17:05:40.32596185Z caller=basic_lifecycler.go:242 msg="instance removed from the ring" ring=store-gateway
level=error ts=2022-11-07T17:05:40.326061756Z caller=mimir.go:774 msg="module failed" module=store-gateway err="invalid service state: Failed, expected: Running, failure: initial blocks synchronization: The specified key does not exist."
level=info ts=2022-11-07T17:05:40.326285533Z caller=module_service.go:114 msg="module stopped" module=runtime-config
level=info ts=2022-11-07T17:05:40.326331402Z caller=memberlist_client.go:641 msg="leaving memberlist cluster"
level=info ts=2022-11-07T17:05:41.483272352Z caller=module_service.go:114 msg="module stopped" module=memberlist-kv
level=info ts=2022-11-07T17:05:41.48377099Z caller=server_service.go:55 msg="server stopped"
level=info ts=2022-11-07T17:05:41.483802931Z caller=module_service.go:114 msg="module stopped" module=server
level=info ts=2022-11-07T17:05:41.483846468Z caller=module_service.go:114 msg="module stopped" module=usage-stats
level=info ts=2022-11-07T17:05:41.483897898Z caller=module_service.go:114 msg="module stopped" module=sanity-check
level=info ts=2022-11-07T17:05:41.487241495Z caller=module_service.go:114 msg="module stopped" module=activity-tracker
level=info ts=2022-11-07T17:05:41.487557035Z caller=mimir.go:763 msg="Application stopped"
level=error ts=2022-11-07T17:05:41.487639699Z caller=log.go:60 msg="error running application" err="failed services\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/__w/mimir/mimir/pkg/mimir/mimir.go:809\nmain.main\n\t/__w/mimir/mimir/cmd/mimir/main.go:212\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"

I use AWS S3 buckets and I am able to write to them.

Has anybody a hint, which key is missing ?

bye, Stefan

[stehnes]

After I changed the s3 configuration, the store-gateway is working now.

Old configuration:

# -- Additional structured values on top of the text based 'mimir.config'. Applied after the text based config is evaulated for templates. Enables adding and modifying YAML elements in the evaulated 'mimir.config'.

structuredConfig:
  alertmanager_storage:
    s3:
      bucket_name: mimir-alertmanager
      access_key_id: AKIA.......................................Q
      endpoint: mimir-ruler.s3.eu-central-1.amazonaws.com
      secret_access_key: R...............................................................T
  blocks_storage:
    backend: s3
    s3:
      bucket_name: mimir-blocks
      access_key_id: AKIA......................................G
      endpoint: mimir-blocks.s3.eu-central-1.amazonaws.com
      secret_access_key: d......................................................S
  ruler_storage:
    s3:
      bucket_name: mimir-ruler
      access_key_id: AKIA......................................Q
      endpoint: mimir-ruler.s3.eu-central-1.amazonaws.com
      secret_access_key: R.....................................................T

New configuration:

# -- Additional structured values on top of the text based 'mimir.config'. Applied after the text based config is evaulated for templates. Enables adding and modifying YAML elements in the evaulated 'mimir.config'.
  structuredConfig:
    alertmanager_storage:
      s3:
        bucket_name: mimir-ruler
        access_key_id: AKIA...........................Q
        endpoint: s3.eu-central-1.amazonaws.com
        secret_access_key: R.......................................................................T
    blocks_storage:
      backend: s3
      s3:
        bucket_name: mimir-blocks........
        access_key_id: AKIA...........................G
        endpoint: s3.eu-central-1.amazonaws.com
        secret_access_key: d......................................................................S
    ruler_storage:
      s3:
        bucket_name: mimir-ruler
        access_key_id: AKIA..........................Q
        endpoint: s3.eu-central-1.amazonaws.com
        secret_access_key: R.....................................................................T

But why I got this message with the old configuration:
evel=info ts=2022-11-07T17:05:35.252927055Z caller=sanity_check.go:44 msg="Object storage config successfully checked"

Does this mean, that the bucket config looks good, but is nevertheless not accessible ?

[dimitarvdimitrov]

thank you for sharing the fix. The logic which checks the object store config tries to get an object that does not exist and expects to receive a 404 from the bucket. That works well to diagnose access control error, but it fails when the bucket returns a 404 because the endpoint always returns a 404. I've opened #3410 to address this issue.