Service telemetry address is not exposed on the collector pod and collector-monitoring service

[pavolloffay]

Component(s)

collector

What happened?

Description

The service.teleemtry.metrics is not properly handled by the operator and the metrics port is not exposed on the operator pod and in the collector-monitoring service.

     telemetry:
        metrics:
          readers:
            - pull:
                exporter:
                  prometheus:
                    host: 0.0.0.0
                    port: 3333

In this case the port 8888 is exposed which is incorrect and not active.

Steps to Reproduce

kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: simplest
  namespace: ploffay
spec:
  observability:
    metrics:
      enableMetrics: true
  deploymentUpdateStrategy: {}
  config:
    exporters:
      prometheus:
        endpoint: "0.0.0.0:9999"
      debug: {}
    processors:
      batch:
        send_batch_size: 10000
        timeout: 10s
      memory_limiter:
        check_interval: 1s
        limit_percentage: 75
        spike_limit_percentage: 15
    receivers:
      jaeger:
        protocols:
          grpc: {}
      otlp:
        protocols:
          grpc:
            endpoint: '0.0.0.0:4317'
          http:
            endpoint: '0.0.0.0:4318'
    service:
      pipelines:
        traces:
          exporters:
            - debug
          processors:
            - memory_limiter
            - batch
          receivers:
            - otlp
        metrics:
          exporters:
            - prometheus
          receivers:
            - otlp
      telemetry:
        metrics:
          readers:
            - pull:
                exporter:
                  prometheus:
                    host: 0.0.0.0
                    port: 3333
EOF

The following ports are exposed on the collector

      ports:
        - name: metrics
          containerPort: 8888
          protocol: TCP
        - name: otlp-grpc
          containerPort: 4317
          protocol: TCP
        - name: otlp-http
          containerPort: 4318
          protocol: TCP
        - name: prometheus
          containerPort: 9999
          protocol: TCP

and -monitoring service

  ports:
    - name: monitoring
      protocol: TCP
      port: 8888
      targetPort: 8888

Expected Result

exposed 3333 port on the collector pod and monitoring service.

Actual Result

Kubernetes Version

1.32

Operator version

main/HEAD

Collector version

main/head

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

Log output

Additional context

No response

[iblancasa]

#4049

[adithya-r-nathan]

Hi @iblancasa,
I am getting the below error when upgrading the collector version to v0.130.0

Error message: “service.telemetry.metrics” decoding failed due to the following error(s): “* has invalid keys: address”
Operator version: v0.85.0
Collector version: v0.130.0

Config:

pipelines:
  logs:
    exporters:
      - debug
      - clickhouse
    processors:
      - memory_limiter
      - transform/vpc_logs
      - batch
    receivers:
      - awss3
telemetry:
  metrics:
    address: 0.0.0.0:8888

Any work around for this?

[iblancasa]

Hi @iblancasa, I am getting the below error when upgrading the collector version to v0.130.0
 Error message: “service.telemetry.metrics” decoding failed due to the following error(s): “* has invalid keys: address”
Operator version: v0.85.0
Collector version: v0.130.0 
Config:

pipelines:
  logs:
    exporters:
      - debug
      - clickhouse
    processors:
      - memory_limiter
      - transform/vpc_logs
      - batch
    receivers:
      - awss3
telemetry:
  metrics:
    address: 0.0.0.0:8888

Any work around for this?

That configuration is not valid anymore: open-telemetry/opentelemetry-collector#11205

[adithya-r-nathan]

@iblancasa Also when I give this

service:
  telemetry:
    metrics:
      level: "basic"
      readers:
        - pull:
            exporter:
              prometheus:
                host: "localhost"
                port: 8888

The address:0.0.0.0:8888 is also present
Config present after updating

service:
  telemetry:
    metrics:
      address: 0.0.0.0:8888
      level: "basic"
      readers:
        - pull:
            exporter:
              prometheus:
                host: "localhost"
                port: 8888

The error still persists

[iblancasa]

That is a different issue. Not related to this one. Please, create a new ticket.

[purkhusid]

Also encountering this issue. This breaks users that have more than one Daemonset of OTEL collectors running since they all try to use port 8888 as a host port.

[iblancasa]

Also encountering this issue. This breaks users that have more than one Daemonset of OTEL collectors running since they all try to use port 8888 as a host port.

You need to configure the .service.telemetry.metrics.readers[x].pull.exporter.prometheus.port field.

It is a different issue than the one explained in the original message. Please, open a different ticket if the previous suggestion does not work for you.

[bgarcial]

@adithya-r-nathan , @purkhusid

The address:0.0.0.0:8888 is also present
Config present after updating ...

Regarding this, you have to reference address:null like this:

metrics:
 address: null
   readers:
     - pull:
         exporter:
           prometheus:
             host: '0.0.0.0'
             port: 8888

but only while using an OpenTelemetry collector version before 0.123.0. On that version, it was ignored https://github.com/open-telemetry/opentelemetry-collector/releases/tag/v0.123.0

Since you are using 0.130.0 then the format should be simply without the address attribute.
From here

Internal telemetry configuration changes
As of Collector v0.123.0, the service::telemetry::metrics::address setting is ignored. In earlier versions, it could be configured with:

service:
  telemetry:
    metrics:
      address: 0.0.0.0:8888

Other References:

• The change from v0.123.0 Mark telemetry.disableAddressFieldForInternalTelemetry as beta, usage of deprecated address are ignored opentelemetry-collector#12756
• v0.128.0 release change log
  • Mark telemetry.disableAddressFieldForInternalTelemetry as stable opentelemetry-collector#13152

[purkhusid]

It's been a couple of weeks but I'm fairly certain that I tried all combinations and you should also see that it does not work if you use some other port than the default 8888. The default port is always exposed as a container port if you don't include the address field with the non-default port but then the issue is that on newer collectors the collector fails to start if you include the address field due to the breaking changes you referenced.