Support default values for configuration environment variables

[Mario-Hofstaetter]

Is your feature request related to a problem? Please describe.

When using Configuration Environment Variables,

if a variable is NOT SET on otelcol start, the value will be empty and probably break configuration.

Please provide a way to set default values in case ENV variables are not available.

Describe the solution you'd like

Allow definition of a default value if an ENV variable is not set. Via ${VAR:default_value} , example:

processors:
  attributes/example:
    actions:
      - key: "${DB_KEY:CustomerDefaultKey}"
        action: "${OPERATION:CustomerDefaultOperation}"

This mimics the syntax of grafana promtail

Another example is docker-compose , where "Default values can be defined inline using typical shell syntax"
https://github.com/docker/docker.github.io/blob/db650b4772f605683ba4143d0b0de44d0a187db4/compose/compose-file/index.md#interpolation

• ${VARIABLE:-default} evaluates to default if VARIABLE is unset or
  empty in the environment.
• ${VARIABLE-default} evaluates to default only if VARIABLE is unset
  in the environment.

and more, which is more complex however.

Describe alternatives you've considered

For example, grafana allows to Override configuration with environment variables

For OTEL Collector, this could look like this. Given this configuration file (section):

exporters:
  jaeger:
    endpoint: my_default_server:14250

Providing an environment variable crafted like so:

export OTELCOL_EXPORTERS_JAEGER_ENDPOINT=some_other_server:23456

Could overwrite the value from default configuration or configuration files.

(Grafana is supporting even more variable substituation ways, but this is certainly overkill for OTEL Collector.)

At the moment, the configuration file (including ENV vars within the file) is the only way to configure OTEL Collector (?), which may be a deliberate design choice. It is not possible to overwrite specific configuration keys by using:

• CLI Flags
• ENV variables directly addressing those keys (so described approach above)

Which is not bad by all means, since mixing all those possiblities creates complex and confusing configuration.

However ENV variables and CLI Flags are easier to use when running OtelCol as a container, in comparison to a config file. When not running on kubernetes opentelemetry-operator is no alternative.

[bogdandrutu]

It is not possible to overwrite specific configuration keys by using:

This is wrong, look into "yaml:" provider (https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/mapprovider/yamlmapprovider/mapprovider.go) you can do something like:

--config=file:my_file --config="yaml:service::telemetry::metrics::address: localhost:8000"

Or using the --set flag.

Allow definition of a default value if an ENV variable is not set. Via ${VAR:default_value}

We will follow the same model as defined by our providers so will be something like ${env:FOO?default=bar} or something like that. I would like to see a PR that changes the envmapprovider to support default value.

[jpkrohling]

This is wrong, look into "yaml:" provider

Is this documented somewhere? This does deserve to be available in user-facing docs. I can write that doc if we have readme files available for all the supported providers.

[neelayu]

@bogdandrutu

We will follow the same model as defined by our providers so will be something like ${env:FOO?default=bar} or something like that. I would like to see a PR that changes the envmapprovider to support default value.

Correct me if I am wrong, but the envmapprovider is basically a replacement for file or yaml map providers, i.e you can have a partial or full configuration stored in an environment variable which you can specify in the --config flag.
What the need is to support the the substitution in expandmapconverter, similar to what we had in #4458

I believe these are two different provisions.
If you agree, I can re-open the previous PR to fit this use-case.

[bogdandrutu]

We are very close to propose for merging #4742 which will use the same providers also inside the config yaml.

[DewaldDeJager]

We are very close to propose for merging #4742 which will use the same providers also inside the config yaml.

I see this PR was merged. Does that mean we can now use this syntax for default values? ${env:FOO?default=bar}

CC @mx-psi

[mx-psi]

We are very close to propose for merging #4742 which will use the same providers also inside the config yaml.

I see this PR was merged. Does that mean we can now use this syntax for default values? ${env:FOO?default=bar}

CC @mx-psi

No, this only means that you can use ${env:FOO} instead of the legacy ${FOO}. We don't support the default= syntax currently.

[ringerc]

See #2534 (comment) for a workaround

It looks like someone did a draft PR at #4458 but it got stalled.

[systems1]

$FOO and ${FOO}` both do not work (exporters)

if they are removed or never included

and ${env:FOO} doesnt work in processors
but $FOO and ${FOO} both work in processors
processors:
attributes:
actions:
- action: insert
key: env
#value: dev # works in all
value: ${env:DEPLOY_ENV} # doesnt work
#value: $DEPLOY_ENV # works
#value: "${DEPLOY_ENV}" # works

tested with otel-col-contriub 0.91.0 and 0.72.0

No, this only means that you can use ${env:FOO} instead of the legacy ${FOO}. We don't support the default= syntax currently.

[mx-psi]

@systems1 this seems like a separate issue, can you report this on a new issue and post a full configuration example that reproduces the issue?

[yurishkuro]

The ticket is 2+ years old, and there is no path forward, #4458 with shell syntax was not accepted.

Is the consensus to go with ${env:VAR?default=xyz} syntax?