Metamonitoring: metrics from mimir cluster are not available

[tagathi]

Hello,
I have installed Grafana Mimir via helm chart, Grafana via helm chart in the same k8s cluster, and have configured a Prometheus that is running in another cluster to remote write to Mimir. It works fine and I can see the Prometheus metrics in Grafana.
I also tried to enable metamonitoring in order to monitor the cluster on which Mimir is installed, by using the below configuration in values.yaml:

metaMonitoring:
  dashboards:
    enabled: true
  serviceMonitor:
    enabled: true
    
  grafanaAgent:
    enabled: true
    installOperator: true

    metrics:
      enabled: true
      remote:
        url: 
        auth:
          username: admin
          passwordSecretName: "agent-secret"
          passwordSecretKey: "password"

      scrapeK8s:
        enabled: true

And I don't seem to get any metrics from this cluster in Grafana. Am I missing some step?
In Grafana operator pod I see this error:

level=error ts=2023-05-23T12:31:57.69149559Z component=controller.grafanaagent name=dev-mimir-meta-monitoring namespace=dev-mimir msg="error during reconc
iling" err="failed to reconcile statefulset for shard: failed to create statefulset: admission webhook \"webhook.openpolicyagent.org\" denied the request:
 Fails CIS Benchmark - Do not admit privileged containers! 

Could this be related?

[dimitarvdimitrov]

Hi, the grafana agent operator creates DaemonSets for logs collection that must either run as root or have access to the root file system. Unfortunately, this probably makes it incompatible with your admission controller. See these comments in the values file for more info

mimir/operations/helm/charts/mimir-distributed/values.yaml

Lines 2839 to 2859 in 942e16c

	# -- SecurityContext of Grafana Agent pods. This is different from the SecurityContext that the operator pod runs with.
	# The operator pod SecurityContext is configured in the grafana-agent-operator.podSecurityContext value.
	# As of mimir-distributed 4.0.0 the Agent DaemonSet that collects logs needs to run as root and be able to access the
	# pod logs on each host. Because of that the agent subchart is incompatible with the PodSecurityPolicy of the
	# mimir-distributed chart and with the Restricted policy of Pod Security Standards https://kubernetes.io/docs/concepts/security/pod-security-standards/
	podSecurityContext:
	# fsGroup: 10001
	# runAsGroup: 10001
	# runAsNonRoot: true
	# runAsUser: 10001
	# seccompProfile:
	# type: RuntimeDefault
	# -- SecurityContext of Grafana Agent containers. This is different from the SecurityContext that the operator container runs with.
	# As of mimir-distributed 4.0.0 the agent subchart needs to have root file system write access so that the Agent pods can write temporary files where.
	# This makes the subchart incompatible with the PodSecurityPolicy of the mimir-distributed chart.
	containerSecurityContext:
	# allowPrivilegeEscalation: false
	# runAsUser: 10001
	# capabilities:
	# drop: [ALL]

But the error you posted suggests it is about the containers in the StatefulSet, not the DaemonSet. You can try to disable logs collection (metaMonitoring.grafanaAgent.logs.enabled: false) and set the values I pointed to above values to something that will be more compatible with your admission controller.

[tagathi]

Hi,
Thank you for the hint. I disabled logs collection, but the error is still present, although not so often. I will try playing around with the SecurityContext values.

[tagathi]

By upgrading mimir-distributed.4.4.1 and setting containerSecurityContext.allowPrivilegeEscalation=false I can see the mimir metrics in Grafana. I had to create a separate datasource with HTTP header X-Scope-OrgID: metamonitoring

[raweber42]

For others being lost with getting the metrics: The metrics for me all start with cortex_[...], I was looking for metrics with mimir in them. Was working, I just searched for the wrong name!