Seeing "received error while streaming chunks from ingester" after rolling out changes to store-gateway

[ericlee123]

Hello Mimir people,

I'm working on tuning our Mimir deployment to be performant and stable during updates. One part is applying the official recommendation of only taking down 2 instances of store-gateway at a time. (Our replication factor is the default value of 3, and we have 10 instances total.)

However, towards the end of the deployment, I am seeing my querier output errors of "received error while streaming chunks from ingester". The extent of my understanding is that I know the querier searches the ingester in the read path for "young data". This leads me to believe there may be some misconfiguration with our deployment (potentially relevant discussion?). Of all the "advanced" configuration options described in that thread, the only one I see that we specify is ignore_blocks_within: 2h.

In order to fix this, we have needed to redeploy querier, and then some upstream service that kept retrying requests to querier, but that's a longer story I will avoid for now.

Does anyone have any idea of what may be the root cause here? Please let me know what other relevant information I can provide.