persistent queue metadata not preserved (v0.129.0) #13342
Description
Component(s)
exporter/exporterhelper
What happened?
Describe the bug
Headers are not preserved when using file_storage extension. Getting unsuccessful logs during export.
Removing the file_storage extension fixes the problem. So export does happen without persistence.
Steps to reproduce
Run collector with provided config and export to an upstream (like loki in this case).
What did you expect to see?
Expecting to see successful export-logs and data received on the upstream.
Expecting persistent queue to preserve metadata.
What did you see instead?
Getting exporting failed logs, due to authentication issues.
Collector version
v0.129.0
Environment information
Environment
Kubernetes: 1.32
OpenTelemetry Operator: 0.126.0
Loki: 3.5.0
OpenTelemetry Collector configuration
## feature-gates: exporter.PersistRequestContext
extensions:
file_storage:
directory: /var/lib/otelcol/file_storage
create_directory: true
headers_setter:
headers:
- action: upsert
key: X-Scope-OrgID
from_context: X-Scope-OrgID
receivers:
otlp:
protocols:
grpc:
include_metadata: true
processors:
batch:
send_batch_size: 10000
timeout: 10s
metadata_keys:
- "x-scope-orgid"
exporters:
otlphttp:
endpoint: http://loki-distributor.loki.svc.cluster.local:3100/otlp
auth:
authenticator: headers_setter
sending_queue:
storage: file_storage
queue_size: 12000
sizer: requests
block_on_overflow: true
retry_on_failure:
enabled: true
initial_interval: 5s
max_interval: 60s
max_elapsed_time: 0
multiplier: 2.0
timeout: 30s
service:
extensions: [headers_setter, file_storage]
pipelines:
logs:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp]Log output
error internal/queue_sender.go:58 Exporting failed. Dropping data. {"resource": {"service.instance.id": "c0b4dd0f-0598-4ce1-929f-37c8259333bc", "service.name": "otelcol-k8s", "service.version": "0.129.0"}, "otelcol.component.id": "otlphttp", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "error": "not retryable error: Permanent error: rpc error: code = Unauthenticated desc = error exporting items, request to http://loki-distributor.loki.svc.cluster.local:3100/otlp/v1/logs responded with HTTP Status Code 401", "dropped_items": 102}
go.opentelemetry.io/collector/exporter/exporterhelper/internal.NewQueueSender.func1
go.opentelemetry.io/collector/[email protected]/exporterhelper/internal/queue_sender.go:58
go.opentelemetry.io/collector/exporter/exporterhelper/internal/queuebatch.(*disabledBatcher[...]).Consume
go.opentelemetry.io/collector/[email protected]/exporterhelper/internal/queuebatch/disabled_batcher.go:23
go.opentelemetry.io/collector/exporter/exporterhelper/internal/queue.(*asyncQueue[...]).Start.func1
go.opentelemetry.io/collector/[email protected]/exporterhelper/internal/queue/async_queue.go:47Additional context
Using feature-gate exporter.PersistRequestContext (#13220) which should implement this feature
Tip
React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.