Evaluating rule failed (consistency check failed because some blocks were not queried)

[miladjahandideh]

We have recorded rules with many time series for a 30d time window and when Mimir wants to evaluate them fails. How can we tune it? which parameters shall we increase? or if we need to increase/add more CPU/RAM, to which Pod?

I increased max_fetched_chunks_per_query: 8000000 and that time store-gateway restarted several times as I inspected.

Evaluating rule failed
err="consistency check failed because some blocks were not queried:

Rule is like the below to have a sense:

count(quantile_over_time(0.99, ping_average_response_ms{}[30d]) < 0.8 )

Also, I see these logs in store-gateway

level=warn ts=2022-10-10T18:04:21.444029971Z caller=memcached_client.go:421 name=chunks-cache msg="failed to fetch items from memcached" numKeys=6137 firstKey=subrange:anonymous/01GETNT8FMK2P8M8GQAXA7P13S/chunks/000063:0:16000 err="read tcp 10.x.x.x:49188->10.x.x.x:11211: i/o timeout"

logs of query-frontend:

level=error ts=2022-10-11T12:47:39.596237978Z caller=retry.go:78 user=anonymous traceID=7d6eea155d4c01b9 msg="error processing request" try=2 err="rpc error: code = Code(500) desc = rpc error: code = Canceled desc = context canceled"
level=error ts=2022-10-11T12:47:42.062758042Z caller=retry.go:78 user=anonymous traceID=7d6eea155d4c01b9 msg="error processing request" try=2 err="rpc error: code = Code(500) desc = rpc error: code = Canceled desc = context canceled"
...
level=error ts=2022-10-11T12:41:23.232709812Z caller=frontend_scheduler_worker.go:253 msg="error sending requests to scheduler" err=EOF addr=x:9095
level=error ts=2022-10-11T12:41:23.232711386Z caller=frontend_scheduler_worker.go:253 msg="error sending requests to scheduler" err=EOF addr=x:9095
level=error ts=2022-10-11T12:41:23.232792908Z caller=frontend_scheduler_worker.go:253 msg="error sending requests to scheduler" err=EOF addr=x:9095
level=error ts=2022-10-11T12:41:23.23281147Z caller=frontend_scheduler_worker.go:253 msg="error sending requests to scheduler" err=EOF addr=x:9095

log of querier:

level=error ts=2022-10-11T12:54:36.505371261Z caller=scheduler_processor.go:119 msg="error processing requests from scheduler" err="rpc error: code = Unavailable desc = closing transport due to: connection error: desc = \"error reading from server: EOF\", received prior goaway: code: NO_ERROR" addr=x:9095
level=error ts=2022-10-11T12:55:01.21259246Z caller=scheduler_processor.go:119 msg="error processing requests from scheduler" err="rpc error: code = Canceled desc = context canceled" addr=x:9095
level=error ts=2022-10-11T12:55:11.348877082Z caller=scheduler_processor.go:213 user=anonymous traceID=65ddc84d6a91155e msg="error notifying frontend about finished query" err="rpc error: code = Canceled desc = context canceled" frontend=x:9095
level=error ts=2022-10-11T12:55:11.348930603Z caller=scheduler_processor.go:168 user=anonymous traceID=65ddc84d6a91155e msg="error notifying scheduler about finished query" err=EOF addr=x:9095
level=error ts=2022-10-11T13:01:25.380164187Z caller=scheduler_processor.go:119 msg="error processing requests from scheduler" err="rpc error: code = Unavailable desc = closing transport due to: connection error: desc = \"error reading from server: EOF\", received prior goaway: code: NO_ERROR" addr=x:9095
level=error ts=2022-10-11T13:01:25.380167955Z caller=scheduler_processor.go:119 msg="error processing requests from scheduler" err="rpc error: code = Unavailable desc = closing transport due to: connection error: desc = \"error reading from server: EOF\", received prior goaway: code: NO_ERROR" addr=x:9095

[pracucci]

err="consistency check failed because some blocks were not queried:

Are the store-gateway replicas healthy? Are they going out of memory?

[miladjahandideh]

We run it with 1 replica and no Ram limit but I will check more and provide more logs.

But when store-gateway crashes Its logs:

...

level=warn ts=2022-10-12T08:52:23.136101937Z caller=memcached_client.go:436 name=chunks-cache msg="failed to fetch items from memcached" numKeys=64 firstKey=subrange:anonymous/01GE0WFS1X0Q0811PV556N6NXG/chunks/000060:77568000:77584000 err="failed to wait for turn. Instance: : context canceled"
level=warn ts=2022-10-12T08:52:23.13612858Z caller=memcached_client.go:436 name=chunks-cache msg="failed to fetch items from memcached" numKeys=36 firstKey=subrange:anonymous/01GE0WFS1X0Q0811PV556N6NXG/chunks/000059:485296000:485312000 err="failed to wait for turn. Instance: : context canceled"
panic: send on closed channel

goroutine 640871 [running]:
github.com/minio/minio-go/v7.(*Object).doGetRequest(_, {{0x0, 0x0, 0x0}, 0x0, 0x0, 0x0, 0x0, 0x1, 0x1, ...})
        /__w/mimir/mimir/vendor/github.com/minio/minio-go/v7/api-get-object.go:319 +0xdc
github.com/minio/minio-go/v7.(*Object).Read(0xc06870a000, {0x0?, 0x0?, 0x0?})
        /__w/mimir/mimir/vendor/github.com/minio/minio-go/v7/api-get-object.go:399 +0x248
github.com/thanos-io/thanos/pkg/objstore/s3.(*Bucket).getRange(0xc00130f950, {0x2598168, 0xc0bbe0d8c0}, {0xc02a0a8f80, 0x32}, 0x1160f980, 0x2af80)
        /__w/mimir/mimir/vendor/github.com/thanos-io/thanos/pkg/objstore/s3/s3.go:427 +0x1b0
github.com/thanos-io/thanos/pkg/objstore/s3.(*Bucket).GetRange(0x7ff51edae688?, {0x2598168?, 0xc0bbe0d8c0?}, {0xc02a0a8f80?, 0x1e7a31558cf9ddbb?}, 0x662652353cae3c2c?, 0x0?)
        /__w/mimir/mimir/vendor/github.com/thanos-io/thanos/pkg/objstore/s3/s3.go:444 +0x28
github.com/thanos-io/thanos/pkg/objstore.(*metricBucket).GetRange(0xc00131c540, {0x2598168, 0xc0bbe0d8c0}, {0xc02a0a8f80, 0x32}, 0x0?, 0x0?)
        /__w/mimir/mimir/vendor/github.com/thanos-io/thanos/pkg/objstore/objstore.go:527 +0xbd
github.com/thanos-io/thanos/pkg/objstore.TracingBucket.GetRange({{0x25ab4d8?, 0xc00131c540?}}, {0x25980c0?, 0xc05e1ab880?}, {0xc02a0a8f80, 0x32}, 0xffb?, 0x475d0a?)
        /__w/mimir/mimir/vendor/github.com/thanos-io/thanos/pkg/objstore/tracing.go:50 +0x22c
github.com/grafana/mimir/pkg/storage/tsdb/bucketcache.(*CachingBucket).fetchMissingSubranges.func1()
        /__w/mimir/mimir/pkg/storage/tsdb/bucketcache/caching_bucket.go:420 +0xf5
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /__w/mimir/mimir/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
        /__w/mimir/mimir/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5

[miladjahandideh]

Tnx @pracucci yes the store-gateway is the one which makes the problem. After running a bit it fails for a while and sometimes recovers. in failure time for queries smaller than retention_period Mimir return queries but for longer times that need to fetch from the store-gateway they fail.

err="inconsistent ring tokens information

we have not set memory or CPU limit.

what is the problem with the store-gateway?

level=info ts=2022-10-19T05:14:19.585410853Z caller=gateway.go:288 msg="synchronizing TSDB blocks for all users" reason=periodic
level=warn ts=2022-10-19T05:14:19.636721901Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFP0KSG5010X20XFSERDF32C err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.63677649Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFP7FBKXV8MT6Z9DSMF8RN7E err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.636797566Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPEB2XPFZAW1PNSSRNRSRSE err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.636816557Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPEB2VYRAX8FW4DEH1G61KV err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.636837864Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPG71ZPET6F8EXT9C79E5BY err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.636852651Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPXKNRMY8Z26RN38QTJMEX8 err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.636868826Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFQ2Y8KYCWGD96MPEC7WS5HP err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.636887839Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFQ9SZNWFKHJBR01GNEPT8GZ err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.636907658Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFP0KMBYP5BSK3J2AZJWFX5A err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.636939266Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFP7FBDWEP0GN675E5RAWBZ0 err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.636959125Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPW2HDP54D747ZME9EWQVH0 err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.636974226Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPW28SHCR47JHEBD4C5Q9ZT err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.636983807Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFQ2Y8RHGDZREPRFY5T42YYA err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.636995541Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFP0KBSHAFCPB8QAH4J2FT5A err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637024923Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPEAT9H61R6742X9Z04P59X err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.63704392Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPEB805QYW7W27BNXDZ9VE9 err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637064953Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPN6Z854VDTWE4XDJKE4MG2 err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637082492Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPW2HBYD52RTT0D5XWVJT68 err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637108Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPW2HGGXJPBH26QPTHX3NZA err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637127594Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFQ9T00G10S8NVCN96YKAA1N err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637148305Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFP0KMDPRP8YY97WK1RJWPP7 err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.63717527Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFP0KMGHH39H2D3MR1G79PM8 err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637199033Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPN6SXW2BA1G94D0431NX57 err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637213198Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFNW4W5N5VSQ9JB41J03DXAZ err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637234913Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFP7FBNP3YT12M2ZZDCB796C err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637253771Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPEB30GES0P89K8EF3AQ96G err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637270801Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFQ2Y01HPDFWS08HNK20R6H8 err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637293535Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFQ2YDR4HZPA6PP40XJKVHGQ err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637312773Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFQ9SQ9H8TZBXVY5P02RSBM8 err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637328171Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFP7FBRG0Z4MVYX85DKJCKGX err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637339844Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFP7F31H592WVNCPNS2VW90K err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637366265Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPEB2NW3TN7HH03JYG6R4ZE err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637381384Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPN6T8GXTX8SEZJWVCAHTGN err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637397266Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFP0KM5XWDZW8VV1FVSNPDMV err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637414626Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPA90X8ZQXK5JWMP4QP0BKG err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637426979Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPPWKMST5MHNTR8NW7T4ZWX err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637451946Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPW2PG5KX9AME66KXX1894T err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637468617Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPXVPFKAVDE5FXRJ4T6J59W err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637487916Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFQ2Y8NP67VNPQ8VWK3ESS1Z err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637506796Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFQ4B8X3GG2JSM87Y1MVT5N6 err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637522561Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFP2WSS1QHNRHSQKMG5JTM5B err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637536243Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFP7FGR45E9EMS3RAQZBRNQP err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637560476Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPCXK3GBV2X936YS1910FYB err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637588933Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPN6T3XB6E7G3J9E0412KFM err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637606943Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPN6T5P6QPNAHY2DGDH6TD3 err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637621481Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPN6HHH8QMM823A3PCBCB2S err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637635367Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPW2H5WZ2EXJXZFSWM1DTAW err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:14:19.637645581Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFQ2Y8DX6SK9MC44Y4TQSMBG err="inconsistent ring tokens information"
level=info ts=2022-10-19T05:14:19.637773927Z caller=gateway.go:294 msg="successfully synchronized TSDB blocks for all users" reason=periodic
level=info ts=2022-10-19T05:31:59.536323469Z caller=gateway.go:288 msg="synchronizing TSDB blocks for all users" reason=periodic
level=warn ts=2022-10-19T05:31:59.585270654Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPW28SHCR47JHEBD4C5Q9ZT err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.585323998Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPXVPFKAVDE5FXRJ4T6J59W err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.585334956Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFQ2Y8RHGDZREPRFY5T42YYA err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.58536126Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFP0KM5XWDZW8VV1FVSNPDMV err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.585372719Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFP7FGR45E9EMS3RAQZBRNQP err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.585405902Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFP7FBNP3YT12M2ZZDCB796C err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.5854214Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPG71ZPET6F8EXT9C79E5BY err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.585454979Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPN6SXW2BA1G94D0431NX57 err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.585473447Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFQ9SQ9H8TZBXVY5P02RSBM8 err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.585502779Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFP0KMBYP5BSK3J2AZJWFX5A err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.585520113Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFP2WSS1QHNRHSQKMG5JTM5B err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.585548996Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPN6T3XB6E7G3J9E0412KFM err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.585576592Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFQ2Y01HPDFWS08HNK20R6H8 err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.585607154Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFQ9T00G10S8NVCN96YKAA1N err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.585633186Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPXKNRMY8Z26RN38QTJMEX8 err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.585649307Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFQ2Y8KYCWGD96MPEC7WS5HP err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.585667084Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFQ4B8X3GG2JSM87Y1MVT5N6 err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.585679421Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFP0KBSHAFCPB8QAH4J2FT5A err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.585691726Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFP7F31H592WVNCPNS2VW90K err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.585721616Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFP7FBDWEP0GN675E5RAWBZ0 err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.585742805Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPEB30GES0P89K8EF3AQ96G err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.585757498Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPW2H5WZ2EXJXZFSWM1DTAW err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.5857703Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFQ9SZXNEHV3F2WGZG65N6VE err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.585783519Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFNW4W5N5VSQ9JB41J03DXAZ err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.585796355Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPEB2VYRAX8FW4DEH1G61KV err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.585812163Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPN6T8GXTX8SEZJWVCAHTGN err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.585834327Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFP7FBKXV8MT6Z9DSMF8RN7E err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.585858233Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPW2PG5KX9AME66KXX1894T err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.585877386Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFQ2Y8NP67VNPQ8VWK3ESS1Z err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.585893816Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFQ9T504S1K6BAV48ZTNFZXX err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.585908966Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFP7FBRG0Z4MVYX85DKJCKGX err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.585930663Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPN6Z854VDTWE4XDJKE4MG2 err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.585947949Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPN6T5P6QPNAHY2DGDH6TD3 err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.585973485Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPW2HGGXJPBH26QPTHX3NZA err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.585991345Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPEB2XPFZAW1PNSSRNRSRSE err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.586007595Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPEB805QYW7W27BNXDZ9VE9 err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.586023629Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPN6HHH8QMM823A3PCBCB2S err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.586050043Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFP0KMGHH39H2D3MR1G79PM8 err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.586068636Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPA90X8ZQXK5JWMP4QP0BKG err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.586094372Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPCXK3GBV2X936YS1910FYB err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.586122384Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPEAT9H61R6742X9Z04P59X err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.58614851Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPEB2NW3TN7HH03JYG6R4ZE err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.586164149Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPW2HBYD52RTT0D5XWVJT68 err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.586187597Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFQ2YDR4HZPA6PP40XJKVHGQ err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.586207466Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFQ9SZNWFKHJBR01GNEPT8GZ err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.586230896Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFQ9SZVXRS3E3WW3N700SYZ7 err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.586247598Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFP0KSG5010X20XFSERDF32C err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.586268367Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFP0KMDPRP8YY97WK1RJWPP7 err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.586280346Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPPWKMST5MHNTR8NW7T4ZWX err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.586298188Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFPW2HDP54D747ZME9EWQVH0 err="inconsistent ring tokens information"
level=warn ts=2022-10-19T05:31:59.586310643Z caller=sharding_strategy.go:129 msg="failed to check block owner and block has been excluded because was not previously loaded" block=01GFQ2Y8DX6SK9MC44Y4TQSMBG err="inconsistent ring tokens information"
level=info ts=2022-10-19T05:31:59.586444637Z caller=gateway.go:294 msg="successfully synchronized TSDB blocks for all users" reason=periodic

And also once I saw this on the Grafana:

execution: expanding series: read bucket index: Get "https://s3.x.com/x-mimir-tsdb/anonymous/bucket-index.json.gz": context canceled

I delete the store-gateway pod and after recreating everything gets ok.

[pracucci]

The error inconsistent ring tokens information is an error which signals a bug in the ring implementation. It's the first time I see it happening, so I still don't have any clue. The store-gateway exposes a web page showing the ring status at /store-gateway/ring. If you request that page with curl -H "Accept: application/json" http://<your domain>/store-gateway/ring you will get a JSON instead. Could you share it with me?

Be aware it contains the hostname and IP of your store-gateways. If that's a sensitive information then please redact it, but make sure each redacted store-gateway ID is still unique in the ring (so I will be able to correlate it with tokens).