`thanos_cache_operation_failures_total` fires on `metadata-cache` with `operation="add", reason="not-stored"`

[NRCan-LGariepy]

Summary

We’re seeing an alert being raised because thanos_cache_operation_failures_total increased. This happens quite often during normal browsing in Grafana. The failing labels are consistently operation="add" and reason="not-stored" with backend="memcached". The failures are reported by the mimir-querier component.

We're just trying to figure out if this is
a) normal
b) abnormal but due to our config
c) a bug in mimir

I'm happy to provide more debugging info as needed. Thanks!

Environment

• Chart: mimir-distributed:5.7.0
• Platform: Rancher RKE2

More info

Metric for the failure:

thanos_cache_operation_failures_total{
    backend="memcached",
    component="querier",
    container="querier",
    service="mimir-querier"
    pod="mimir-querier-688b5d66d5-lrvtd",
    endpoint="http-metrics",
    job="mimir-querier",
    name="metadata-cache",
    namespace="mimir",
    operation="add",
    reason="not-stored",
    ...
}

[56quarters]

This should be fixed via #9658 in Mimir 2.15 or later. Where / how is the alert that's firing defined?

[NRCan-LGariepy]

Seems to be coming from this PrometheusRule:

spec:
  groups:
    - name: mimir.rules
      rules:
        - alert: Error or Failure
          annotations:
            description: >-
              The value of the metric {{$labels.metric}} for the component
              {{$labels.job}} as increased in the last 24h (pod {{$labels.pod}}).
            summary: An error or failure occured.
          expr: >-
            sum by (metric, job, pod, namespace, cluster_name)
            (increase(label_replace({__name__=~".*(error|fail.*).*",
            job=~"mimir-(compactor|distributor|ingester|querier|query-frontend|store_gateway)"},
            "metric", "$1", "__name__", "(.*)")[1d:])) > 0
          labels:
            email: '""'
            issue: Error or failure
            severity: critical

Although this is in the cattle-monitoring-system namespace, so it might have to do with how mimir is packaged for Rancher, or something.

[NRCan-LGariepy]

@56quarters If this is not an alert that Mimir itself is distributing, feel free to close the issue.

[56quarters]

If the alert were coming from the Mimir mixin it should be called MimirCacheRequestErrors so it does sound like it's coming from something else. I'll close for now and you can reopen if you find that it's coming from Mimir. Thanks!