Configuration sample for Distributor Forwarding in YAML format

[LarsMilland]

Hi

I am trying to wrap my head around how to one configures the forwarding of metrics from the Distributor component.

The functionality that was implemented here:
64b1d9c

I am looking for a sample in YAML format that would take a list of metrics send via remote write to a tenant for the Distributor to store in Mimir, where the Distributor would also forward that list of metrics to another remote write enabled Prometheus system.

The setup would need to support "Bearer token" as the authentication means towards the remote Prometheus endpoint if that is possible, and show how to configure which metrics to forwards in addition to storing locally in Mimir and of course show how to configure the remote Prometheus endpoint with credential for the authentication part.

I very much hope this is possible - and would be very grateful if anyone would have such a sample or could construct such a sample.

best regards
Lars Milland

[pstibrany]

Hi. Right now the forwarding feature has no support for adding authentication headers.
As for configuration, it needs to be stored in runtime configuration, and looks something like this:

    overrides:
        "tenant":
            forwarding_drop_older_than: 1m
            forwarding_endpoint: http://remote_host/api/v1/write
            forwarding_rules:
                cortex_compactor_meta_sync_failures_total:
                    ingest: false
                cortex_discarded_exemplars_total:
                    ingest: false

Each of the mentioned metrics will be forwarded to configured endpoint.

Note that this feature is experimental, and very likely to be removed in the future.

[pstibrany]

I'm curious, what is your use case for using forwarding?

[LarsMilland]

Many thanks for showing the example.

Our usecase is using Mimir as a fine-grained and long-term metrics backend, and then we have some other more "monitor of monitor" based platform that supports Service Level Availability dashboarding from several other sources, where we would like to have a subset of Mimir ingested metrics forwarded to.