insecure_skip_verify=true is not working while sending metric data in datadog exporter #40085
Description
Component(s)
exporter/datadog
What happened?
Description
I'm experiencing a TLS error similar to the one discussed in issue #22772.
We have an Azure Container App that has been running otel/opentelemetry-collector-contrib:latest. Metrics were publishing to Datadog as expected until ~24 hours ago. Since then, metrics have stopped being exported, while logs and traces continue to flow normally.
The error:
2025-05-15T01:39:54.515Z error [email protected]/zaplogger.go:67 TLS Handshake failure: tls: failed to verify certificate: x509: certificate signed by unknown authority {"resource": {}, "otelcol.component.id": "datadog", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics"}
2025-05-15T01:39:54.515Z error [email protected]/zaplogger.go:67 Error while processing transaction: error while sending transaction, rescheduling it: Post "https://api.us3.datadoghq.com/api/v2/series": tls: failed to verify certificate: x509: certificate signed by unknown authority {"resource": {}, "otelcol.component.id": "datadog", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics"}
We already have insecure_skip_verify: true configured in our collector config, which makes this error unexpected.
Steps to Reproduce
This issue is only happening in our Azure Container App environment. Running the same image and config in local Docker does not reproduce the issue. Metrics export successfully in that case.
Expected Result
Metrics published in Datadog.
Actual Result
TLS handshake failure
Collector version
0.126.0
Environment information
Environment
OS: Linux
OpenTelemetry Collector configuration
receivers:
otlp:
protocols:
grpc:
endpoint: "0.0.0.0:4317"
http:
endpoint: "0.0.0.0:4318"
filelog:
include: [ /etc/logs/app/*]
exclude_older_than: 120h
start_at: end
operators:
- type: json_parser
processors:
batch:
timeout: 10s
attributes:
actions:
- key: env
value: prod
action: upsert
exporters:
debug:
verbosity: detailed
file:
path: /etc/logs/traces.json
format: json
flush_interval: 5s
datadog:
api:
site: us3.datadoghq.com
key: ""
logs:
endpoint: http://http-intake.logs.us3.datadoghq.com
tls:
insecure_skip_verify: true
service:
telemetry:
metrics:
address: ""
level: "none"
logs:
level: "info"
pipelines:
traces:
receivers: [otlp]
processors: [attributes, batch]
exporters: [debug, file, datadog]
logs:
receivers: [filelog]
processors: [batch]
exporters: [datadog]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [datadog]Log output
2025-05-15T01:39:54.515Z error [email protected]/zaplogger.go:67 TLS Handshake failure: tls: failed to verify certificate: x509: certificate signed by unknown authority {"resource": {}, "otelcol.component.id": "datadog", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics"}
2025-05-15T01:39:54.515Z error [email protected]/zaplogger.go:67 Error while processing transaction: error while sending transaction, rescheduling it: Post "https://api.us3.datadoghq.com/api/v2/series": tls: failed to verify certificate: x509: certificate signed by unknown authority {"resource": {}, "otelcol.component.id": "datadog", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics"}Additional context
No response