HTTP 400 "proto: wrong wireType = 1 for field BucketCounts" error after updating to v0.133.0

[martincostello]

Component(s)

exporter/otlphttp

What happened?

Describe the bug

After updating from v0.132.2 to v0.133.0 in grafana/docker-otel-lgtm as part of our v0.11.9 release, applications submitting metrics to http://${host}:4318/v1/metrics with the following configuration receive an HTTP 400 error with the following response:

OTEL_EXPORTER_OTLP_ENDPOINT=http://${host}:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

proto: wrong wireType = 1 for field BucketCounts

Steps to reproduce

I don't have a minimal reproduction, but grafana/docker-otel-lgtm@af35c63 shows a change to our LGTM acceptance tests that fail when the configuration is changed to the above for a .NET application instead of sending gRPC to http://${host}:4317.

What did you expect to see?

Metrics are successfully posted to the endpoint.

What did you see instead?

HTTP 400 response with the response:

proto: wrong wireType = 1 for field BucketCounts

Collector version

v0.133.0

Environment information

Environment

OS: Ubuntu 24.04.3
Compiler: N/A

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
        cors:
          allowed_origins:
            - http://*
  prometheus/collector:
    config:
      scrape_configs:
        - job_name: "opentelemetry-collector"
          scrape_interval: 1s
          static_configs:
            - targets: ["127.0.0.1:8888"]

extensions:
  health_check:
    endpoint: 0.0.0.0:13133
    path: "/ready"

processors:
  batch:

exporters:
  otlphttp/metrics:
    endpoint: http://127.0.0.1:9090/api/v1/otlp
    tls:
      insecure: true
  otlphttp/traces:
    endpoint: http://127.0.0.1:4418
    tls:
      insecure: true
  otlphttp/logs:
    endpoint: http://127.0.0.1:3100/otlp
    tls:
      insecure: true
  otlp/profiles:
    endpoint: http://127.0.0.1:4040
    tls:
      insecure: true
  debug/metrics:
    verbosity: detailed
  debug/traces:
    verbosity: detailed
  debug/logs:
    verbosity: detailed

service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/traces]
      #exporters: [otlphttp/traces,debug/traces]
    metrics:
      receivers: [otlp, prometheus/collector]
      processors: [batch]
      exporters: [otlphttp/metrics]
      #exporters: [otlphttp/metrics,debug/metrics]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/logs]
      #exporters: [otlphttp/logs,debug/logs]
    profiles:
      receivers: [otlp]
      exporters: [otlp/profiles]

Log output

Additional context

No response

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[martincostello]

Possibly introduced by #13626.

[mx-psi]

cc @bogdandrutu @dmitryax

[jade-guiton-dd]

I've reproduced this issue with a custom client. It seems to be a compatibility issue in the new protobuf parser introduced in #13626, which expects all repeated primitive fields (such as the bucket_counts field) to be encoded using the packed format, even though the spec states:

Protocol buffer parsers must be able to parse repeated fields that were compiled as packed as if they were not packed, and vice versa.

So any client:

• using a protobuf serializer which (for some reason) does not emit the packed format for those fields
• who emits histograms / exponential histograms / profiles

will receive an error starting from 0.133.0.

[mx-psi]

@martincostello It would be helpful to have a test (preferrably a unit test) that can reproduce this issue

[martincostello]

Unfortunately this is many levels down the stack from the level I found this bug in as an end-user, so I'm not sure where I would need to start to do that (especially if you want it in Go).

The referenced commit in the issue description does reproduce the issue, it's just an end-to-end test with many different moving parts (a .NET application using the OpenTelemetry .NET SDK, Docker Compose, our LGTM image containing the collector, and our OATs test tool).

[jade-guiton-dd]

It would be helpful to have a test (preferrably a unit test) that can reproduce this issue

Here's a raw HTTP payload in hex format that triggers the issue:

0a1b121912170a0874657374686973744a0b0a09310100000000000000

It contains a barebones Histogram, where the bucket_counts are encoded in the unpacked format. (It's not very realistic since it's missing every field except that one, but I'd expect any additional validation to be past the parsing stage anyway.)

It needs to be POSTed to /v1/metrics, with Content-Type: application/x-protobuf.

For manual testing, here's a cURL command:

echo -en '\n\x1b\x12\x19\x12\x17\n\x08testhistJ\x0b\n\t1\x01\x00\x00\x00\x00\x00\x00\x00' | \
    curl -s -H 'Content-Type: application/x-protobuf' --data-binary @- -o - http://localhost:4318/v1/metrics | \
    xargs -0 printf '%q\n'

Output with Collector 0.132.0:

''$'\n'

Output with Collector 0.133.0:

''$'\b\003\022''0proto: wrong wireType = 1 for field BucketCounts'

[jade-guiton-dd]

Looks like this bug can be worked around by disabling the pdata.useCustomProtoEncoding feature gate, which can be done by adding --feature-gates -pdata.useCustomProtoEncoding to the command line arguments used to start the Collector.

[bogdandrutu]

Per our proto definition see https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/metrics/v1/metrics.proto#L487 and https://protobuf.dev/programming-guides/proto3/ (the part about packed)

packed (field option): Defaults to true on a repeated field of a basic numeric type, causing a more compact [encoding](https://protobuf.dev/programming-guides/encoding#packed) to be used. To use unpacked wireformat, it can be set to false. This provides compatibility with parsers prior to version 2.3.0 (rarely needed) as shown in the following example:

repeated int32 samples = 4 [packed = false];

Based on that, it is not expected to receive unpacked data.