Attributes for component instancing

[djaglowski]

Is your feature request related to a problem? Please describe.

Component instancing is complicated. Since it seems unlikely that we will change this prior to 1.0, I want to raise the broader discussion of how the collector's own telemetry should ascribe telemetry to component instances.

First, why should we ascribe telemetry to component instances rather than to component configurations? As an observability tool it is critical that we are a good exemplar of an observable system. If we cannot accurately represent the internal state of the collector, I believe we will have failed to accomplish this goal.

In practice, what is most important is that instances derived from the same configuration may have very different states.

Describe the solution you'd like

Many components utilize global state to override the default instancing behavior (i.e. sharedcomponent or similar), but to keep things simple, I will first propose an attribution schema which ignores this problem.

Ignoring shared instances

The following schema provides sufficient information to ascribe telemetry to specific instances. Since each kind of pipeline component (receivers, processors, exporters, connectors) has its own instancing rules, there are separate sets of attributes proposed for each kind.

Receivers

• receiver: The component ID
• otel.signal: logs, metrics, or traces

Processors

• processor: The component ID
• otel.signal: logs, metrics, or traces
• pipeline: The pipeline ID

Note: otel.signal is redundant because pipeline contains the same info. Is it worth keeping for consistency with receivers and exporters, even if it is not consistent with connectors?

Exporters

• exporter: The component ID
• otel.signal: logs, metrics, or traces

Connectors

Note: otel.signal as used in the other component kinds is insufficient even for metrics which describe ingoing or outgoing telemetry. This is because multiple instances of a connector may have the same type of incoming or outgoing telemetry. Moreover, the set of attributes should be sufficient for all telemetry describing the instance. This leaves two options:

1. Separate incoming and outgoing attributes:

• connector: The component ID
• otel.incoming_signal: logs, metrics, or traces
• otel.outgoing_signal: logs, metrics, or traces

2. otel.signal with special values:

• connector: The component ID
• otel.signal: logs->logs, logs->metrics, logs->traces, metrics->logs, metrics->metrics, etc

Acknowledging shared instances

As far as I am aware, all cases of shared instances are implemented using a singleton pattern. The following schema assumes that this is universally the case.

Receivers

• receiver: The component ID
• otel.signal: logs, metrics, traces, OR ALL

Processors

• processor: The component ID
• otel.signal: logs, metrics, traces, OR ALL
• pipeline: The pipeline ID, OR ALL

Exporters

• exporter: The component ID
• otel.signal: logs, metrics traces, OR ALL

Connectors

1. Separate incoming and outgoing attributes:

• connector: The component ID
• otel.incoming_signal: logs, metrics, OR ALL
• otel.outgoing_signal: logs, metrics, OR ALL

2. otel.signal with special values:

• connector: The component ID
• otel.signal: logs->logs, logs->metrics, logs->traces, metrics->logs, metrics->metrics, etc, OR ALL

Acknowledging non-singleton shared instances

It is worth noting that there is nothing that explicitly prevents component authors from overriding instancing with non-singleton patterns. Deriving some examples from existing components, one could imagine an otlp receiver which allows users to assign signals to ports so that e.g. logs and traces are accepted on one port while metrics are accepted on another. Another example is a memory limiter processor which limits memory across only a single signal type (so would need to be shared across multiple pipelines, but not all pipelines).

I believe the solution to handling this would have to be based on a "list of otel.signal" attribute:

• otel.signals: one or more values which can be assigned to otel.signal

Describe alternatives you've considered

Additional context
Work on processor metrics has progressed recently but when considering which attributes to apply to processors, it raises the question of whether the same attributes may apply to other component kinds. For example, the recently added otel.signal attribute may be useful for receivers and exporters, but insufficient for connectors.

[djaglowski]

My preference is to acknowledge shared components, but assume they are singleton instances. For connectors, I prefer otel.signal with a special set of values. For processors, I prefer keeping both pipeline and otel.signal (so that the latter is universal across component kinds).

Receivers

• receiver: The component ID
• otel.signal: logs, metrics, traces, OR ALL

Processors

• processor: The component ID
• otel.signal: logs, metrics, traces, OR ALL
• pipeline: The pipeline ID, OR ALL

Exporters

• exporter: The component ID
• otel.signal: logs, metrics traces, OR ALL

Connectors

• connector: The component ID
• otel.signal: logs->logs, logs->metrics, logs->traces, metrics->logs, metrics->metrics, etc, OR ALL

[mx-psi]

After discussing on the 2024-09-18 Collector stability meeting, I support this schema

[djaglowski]

Closing in favor of #11343