[exporter/prometheusremotewrite] Support Prometheus Remote-Write v2

[jmichalek132]

Component(s)

exporter/prometheusremotewrite

Is your feature request related to a problem? Please describe.

Prometheus remote write v2 spec is close to being finalized, it would be good time to start working on implementing support for it in the prometheus remote write exporter component. Soon backend such as prometheus, mimir, thanos etc will start supporting it. It would great to start working on implementing to gain it's benefits,.

Describe the solution you'd like

I would like to contribute implementation of the remote write v2 prometheus/docs#2462 into the prometheus remote write exporter, most likely behind a feature flag since long-term v2 version is supposed to replace the current implementation.

Describe alternatives you've considered

Potentially it could introduced as a separate component but I hope that won't be necessary.

Additional context

The current implementation of remote write v2 lives in it's own branch of prometheus, https://github.com/prometheus/prometheus/tree/remote-write-2.0 , but should be merged soon.

TODO

Translation layer:

• Support for metadata [chore] prom translation rw2 add support for metadata #40469
• Handle conflicts in metrics handle conflits in prw v2 #39570
• Handle target_info [chore] prom translation rw2 add support for target_info #40493
• Handle exemplars, seems partially implemented in feat: add support for MetricTypeSum #37156
• Support for metric types
  • MetricTypeSum feat: add support for MetricTypeSum #37156
  • MetricTypeHistogram [chore] prom translation rw2 add support for histogram #40494
    • Add support for writev2.Histogram and enable by default. prom translate rw2 optionally translate histogram to nhcb 2 #42606
    • look into this comment [chore] prom translation rw2 add support for histogram #40494 (comment)
  • MetricTypeExponentialHistogram prom translation rw2 add support for exponential histogram #42267
  • MetricTypeSummary Jm prom translation rw2 add support for summaries #40750
  • Double check this logic, it's not present in RW1

    opentelemetry-collector-contrib/pkg/translator/prometheusremotewrite/metrics_to_prw_v2.go

    Line 98 in 1af0f49

    if !metric.Sum().IsMonotonic() {

• Figure out what to do with namespace added to target_info in both RW1 and RW2 [chore] prom translation rw2 add support for target_info #40493 (comment)

Exporter:

• Get initial PR merged http://github.com/open-telemetry/opentelemetry-collector-contrib/pull/38235
• Sort timeseries https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/399805
• Implement batching [chore] prom rw v2 exporter add support for batching #40051
  • batch symbolsTable instead of sending it whole
  • Try sorting timeseries while batching??
  • Check if sizeOfSeries > maxBatchByteSize could happen, this might affect PRW1 too [chore] prom rw v2 exporter add support for batching #40051 (comment)
• Add test case with invalid proto message [exporter/prometheusremotewrite] feat: prom rw exporter add support for rw2 #35888 (comment) [chore] prom rw exporter add test case for invalid proto #40562
• Log warning when missing RW2 spec headers [chore] prom rw exporter add extra logs for the RW2 code path #40559
• Implement WAL support
• Go over the spec and confirm it follows it
• Add support for NoTranslation mode (similar to [Prometheus] Exporter should support Prometheus translation modes opentelemetry-go#6668)
• Look into generating metrics from Written headers Prom rw exporter, rw2 generate metrics from written headers #40866
• Add more detailed metrics around failure [exporter/prometheusremotewrite] Add Detailed Export Failure Metrics #39799 (comment)
• Improve tests

  opentelemetry-collector-contrib/exporter/prometheusremotewriteexporter/exporter_test.go

  Line 477 in c9e3f1a

  // TODO check labels

• Optionally add e2e tests?
• Add testing of otel inprometheus/compliance repo remote_write_sender: Add support for Remote Write 2.0 prometheus/compliance#101

Linking upstream Prometheus issue for RW2 stability
prometheus/prometheus#16944

Snippet for later

			symbols := converter.symbolTable.Symbols()
			for _, tss := range converter.timeSeries() {
				lbls := make([]prompb.Label, 0, len(tss.LabelsRefs)/2)
				for i := 0; i < len(tss.LabelsRefs); {
					lbls = append(lbls, prompb.Label{Name: symbols[tss.LabelsRefs[i]], Value: symbols[tss.LabelsRefs[i+1]]})
					i += 2
				}
				assert.Equal(t, tc.wantLabels, lbls)
			}

[github-actions]

Pinging code owners:

• exporter/prometheusremotewrite: @Aneurysm9 @rapphil

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

Thanks for the proposal @jmichalek132, makes sense to me. Agreed that a feature gate is a good idea here. 👍

[varshith257]

Interesting! How can I contribute to this feature? @crobert-1 ,@jmichalek132

[rapphil]

Remote write 2.0 is still experimental in prometheus, however it will be enabled by default in the the upcoming version version 2.54.0.

Having said that I would welcome a contribution that will add support to PRW2.0. I think we can add a configuration option protocol_version that can be used by users to select what version of the remote write protocol they want to use. Valid values would be 1.0 and 2.0 with default value 1.0. Once Prometheus deprecate 1.0 we can plan on moving the default to 2.0.

[varshith257]

Thanks @rapphil . Will look into the Prometheus remote write 2.0 and plan the changes with above said

[dashpole]

cc @ArthurSens
We were considering making this a CNCF intern project: cncf/mentoring#1281