Bug: Ruler seeing errors while streaming chunks from ingester for a specific tenant

[bhargavmg]

What is the bug?

We are seeing the below errors in the ruler pods which seem to indicate issues loading chunks from ingester for a specific tenant only.

caller=streaming.go:109 method=SeriesChunksStreamReader.StartBuffering user=test level=error msg="received error while streaming chunks from ingester" err="rpc error: code = Canceled desc = context canceled"

The logs are not very helpful. We tried rolling the ruler pods and verified that ingesters are healthy as well. There seems to be no issues with the configurations since it was working fine previously and the issue started happening all of a sudden without any changes to the config.

How to reproduce it?

There hasn't been any changes we did on our end to suggest the best possible way to reproduce the error. The only change we did was upgrade Mimir to 2.15.0 from 2.13.0 and the issue started happening 4 days after the upgrade for a specific tenant only.

What did you think would happen?

We noticed similar errors once in 2.13,0 version and rolling the ruler pods fixed the issue. Expected these errors to be resolved in the latest version or atleast resolved during a restart of ruler pods.

What was your environment?

Infrastructure: Kubernetes
Deployment: Helm
Mimir 2.15.0

Any additional context to share?

No response

[charleskorn]

Do you see any queries failing? Or are you just seeing that message logged but all queries are evaluating successfully?

[bhargavmg]

I dont see queries failing as such. But I dont see any outputs for the error budgets for the specific tenant as well. Also, if I look at the metrics dashboard, I see the ruler's SLO dropping specifically in rule evaluations for that single tenant. I'm having a hard time understanding what chunks are failing to be fetched from ingester for the ruler

[bhargavmg]

@charleskorn Can I get some help on this please

[charleskorn]

If all you're seeing is that error message being logged, but queries are evaluating without problems, then this message is safe to ignore: it just indicates that the request was cancelled.