[exporter/datadog]: `fail_on_invalid_key` is not respected when API key contains disallowed characters

[ste93cry]

Component(s)

exporter/datadog

What happened?

Description

The Datadog exporter offers a configuration option named fail_on_invalid_key, designed to prevent startup failures when an invalid key is used. If the key fails authentication with the provider, the collector logs the error and proceeds to start successfully. However, commit fae5b26 introduced an additional check to validate the key’s format. This new check does not honor such setting, causing the collector to crash during startup if the API key contains invalid characters.

Steps to Reproduce

Configure the datadog exporter with a key that does not respect the allowed pattern [^a-fA-F0-9] and set fail_on_invalid_key: false

Expected Result

The collector starts successfully, logging the error and ignoring it

Actual Result

The collector crashes with a fatal error during startup time

Collector version

v0.135.0

Environment information

Environment

OS: MacOs

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

exporters:
  datadog:
    api:
      fail_on_invalid_key: false
      site: datadoghq.eu
      key: test-api-key

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [datadog]

Log output

Error: invalid configuration: exporters::datadog: api::key contains invalid characters: invalid characters: t, s, t, -, p, i, -, k, y
2025/09/14 20:18:33 collector server run finished with error: invalid configuration: exporters::datadog: api::key contains invalid characters: invalid characters: t, s, t, -, p, i, -, k, y

Additional context

No response

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Pinging code owners:

• exporter/datadog: @mx-psi @dineshg13 @liustanley @songy23 @mackjmr @ankitpatel96 @jade-guiton-dd @IbraheemA

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[mx-psi]

@ste93cry What is the use case for this?

[ste93cry]

I actually have two use cases for this. The first relates to consistency: as a user, I was surprised to discover that setting fail_on_invalid_key to false didn’t prevent failures when the key format itself was invalid. I couldn’t find any documentation explaining this behavior or clarifying that format validation is handled differently. The second use case is the one that truly matters to me: the key used by the collector is supplied by an external party and originates from user input. I was stunned to learn that someone had configured an invalid key, which ended up crashing the entire fleet of collector instances on the server.

[mx-psi]

I actually have two use cases for this. The first relates to consistency: as a user, I was surprised to discover that setting fail_on_invalid_key to false didn’t prevent failures when the key format itself was invalid. I couldn’t find any documentation explaining this behavior or clarifying that format validation is handled differently.

fail_on_invalid_key refers to runtime validation, not static validation. We can clarify this on the documentation if needed.

The second use case is the one that truly matters to me: the key used by the collector is supplied by an external party and originates from user input. I was stunned to learn that someone had configured an invalid key, which ended up crashing the entire fleet of collector instances on the server.

If you use the validate subcommand (e.g. otelcontribcol validate) you can verify that your configuration is valid statically (e.g. you could do this in a CD pipeline right before deployment). Nonetheless, this is a bit more challenging than in other fields since we are talking about a secret. I'd be interested to know if using the validate subcommand would work for your infrastructure/deployment model.

[ste93cry]

fail_on_invalid_key refers to runtime validation, not static validation

Honestly, I’m not convinced there's much value in applying static validation to the API key format. I do agree that structural validation of the config is useful, but validating the actual data and treating that as static validation doesn’t feel quite right to me.

I'd be interested to know if using the validate subcommand would work for your infrastructure/deployment model.

My use case isn’t CI-related. Users configure the OTel Collector and input their Datadog API key via a web page. This config is merged across tenants by a Kubernetes operator and used by a single collector deployed as a DaemonSet. If one tenant’s config is invalid and crashes the collector, it affects all tenants. Deploying a collector per tenant would isolate failures, but since each node needs a collector to collect container logs, this would lead to an unmanageable amount of resource usage.

I’ve added UI validation to prevent saving invalid keys now, but it comes with trade-offs. I only realized the need for it after running into issues, and if the key format ever changes, I won’t know until it’s too late. As I said, I don't think this kind of validation falls into the "static validation" umbrella.

[mx-psi]

I filed #42783 to remove this kind of validation entirely