Prometheus metrics broken on prom 3.0

[ishioni]

Describe the bug
Prometheus 3.0 rejects the scrape because dragonfly does not set content-type

To Reproduce
Steps to reproduce the behavior:

1. curl http://dragonflydb:adminport
2. note the lack of content-type in the http response header
3. Prometheus 3.0 complains with "Error scraping target: non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target"

Expected behavior
Prometheus metrics should work

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• OS: [talos 1.8]
• Kernel: # 6.6
• Containerized?: [Kubernetes]
• Dragonfly Version: [1.25.4]

Additional context
Setting something like
content-type text/plain; version=0.0.4; charset=utf-8; escaping=underscores
should be enough to make it work. Adding a fallback_scrape_protocol should be good as well, but the prometheus operator CRDs don't allow that unfortunately

More context https://prometheus.io/docs/prometheus/3.0/migration/#scrape-protocols

[romange]

@ishioni should be easy to fix. would you like to take a stab on it?

[ishioni]

Wish I could - last time I touched cpp was over a decade ago in college

[lnd-d]

I added this to prometheus.yml meanwhile.

global:
 metric_name_validation_scheme: legacy

[VirtualDisk]

I added this to prometheus.yml meanwhile.

global:
 metric_name_validation_scheme: legacy

How do you enable this in the Prometheus Operator?

[beauwest]

A more targeted solution for now is to apply a fallBackScrapeProtocol to the ServiceMonitor or PodMonitor.

Full example:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: dragonfly-servicemonitor
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: dragonfly
  endpoints:
    - port: service-port
  fallbackScrapeProtocol: PrometheusText1.0.0

[romange]

How do I reproduce it locally? Running the latest version of prometheus is enough?

[beauwest]

Yes sir, running Prometheus 3 is enough to reproduce it. The /metrics endpoint does not return a Content-Type header.

From the migration guide

Prometheus v3 is more strict concerning the Content-Type header received when scraping. Prometheus v2 would default to the standard Prometheus text protocol if the target being scraped did not specify a Content-Type header or if the header was unparsable or unrecognised. This could lead to incorrect data being parsed in the scrape. Prometheus v3 will now fail the scrape in such cases.