Out of Order Metrics

[sarang10]

We are evaluating Mimir for our Metrics TSDB store. We have a high throughput system of order of 5M writes per second, and we use Kafka for collecting all such metrics before pushing into mimir. Lag can built up sometimes in any of the partition of order of 4-6 hours, which might lead to out of order samples.
Currently we don't have a partitioning scheme, but even if we introduce time series based partitioning, out of order metrics can still come with respect to the current timestamp of the TSDB head block.

We know that there is an experimental feature for handling out of order samples through which we can configure Grafana Mimir to set an out_of_order_time_window threshold for how old samples can be ingested. Keeping the out_of_order_time_window flag as 6 hours will lead to increase CPU and memory utilisation in Ingestors.
We can also use backfilling directly bypassing ingestor, through mimirtool for samples older than the out_of_order_window specified.

So is it the right approach or are there any other alternatives to tackle this ?

[sarang10]

@pracucci Can you please help here ?