[receiver/otlp] Review telemetry

[mx-psi]

As part of the stabilization of the OTLP receiver, we want to ensure that it has the core self telemetry needed to understand the component and debug common issues. This issue tracks reviewing existing telemetry and defining any new telemetry needed to make the component observable. It should be rare that we add new telemetry as per the 1.0 tenet that "The Collector is already used in production at scale and has been tested in a variety of environments".

[jade-guiton-dd]

Review of internal telemetry in OTLP components

Instead of handling exporter/otlp, and exporter/otlphttp in their respective issues (#11140 and #11141), I decided to condense everything in one comment for easier reading and comparison, since they have a lot in common.

I mostly focused on metrics, as only those are stable according to the docs.

More details on currently available telemetry can be found in the "Internal telemetry" docs page.

I compared the existing telemetry with the metrics:

• exposed by the Datadog agent;
• exposed by Prometheus;
• exposed by Vector;
• exposed by Fluentbit;
• mentioned in this old PR Add Observability Vision document #188 defining what values we would like to be observable.

Based on this, I listed some ideas for new metrics which we might want to implement before stabilization.

I may update this comment if I find additional ideas in other sources.

Note that most, if not all of these new metrics would not be specific to the 3 components in question, and would probably reside in the exporterhelper/receiverhelper, or in the instrumentation for the gRPC and HTTP client/server libraries.

Related work / dependencies

The work-in-progress pipeline auto-instrumentation RFC may end up defining metrics which collide with the existing telemetry (otelcol.receiver.produced.items with otelcol_receiver_accepted_*). We may want to wait until that RFC is finalized, so we can decide whether to deprecate the older metrics or not. Moreover, it defines attributes for identifying the source of internal telemetry signals, which we may want to utilize as well.

There is also work being done on adding tracing to all components (#8804), which might impact the present issue.

Action items

We will need to decide:

1. which (if any) of these metrics are useful enough to be worth implementing.
2. which (if any) are blocking for stabilization.
3. how to deal with the auto-instrumentation RFC metrics and attributes.

receiver/otlp

Current telemetry

This telemetry is provided by the receiverhelper module and the gRPC/HTTP server libraries; none of it is internal to this component.

• Counter of successes and failures to receive a value, per signal type
• HTTP: request size, request duration, response size, active request count
• gRPC: request size, request duration, response size

Proposals for new metrics

1. Count of active gRPC requests
2. Failure counts per type of error, for example:
   • Network errors
     • Potentially differentiate DNS errors, TLS errors, connection errors, and errors when reading/writing (the Datadog agent does this when exporting)
   • User errors (failure to unmarshall request / wrong method/content type for HTTP)
   • Internal errors (failure to marshall response)
   • Failure further down the pipeline (the only kind of failure which is currently counted I believe!)
     • Potentially differentiate between retryable and non-retryable errors the same way we do in the response status
3. Success counts at each step of the pipeline (probably overkill)
4. Counter of payloads (un)successfully processed (as opposed to individual values)
5. Counter of bytes (un)successfully processed (as opposed to "request size" which does not expose the success state)
6. Counter of compressed bytes processed (as opposed to "request size" which is after decompression)
7. Data processing/dropping rate (as opposed to a counter, in units/s or bytes/s)
8. Connection/request rate (as opposed to a counter)
9. Timing of suboperations (reception, unmarshalling, processing, marshalling, acknowledgement; should we prefer traces and trace-derived metrics over this?)
10. Overall latency through component
11. Total count of requests / connections that opened / closed / errored

exporter/otlp

Current telemetry

This telemetry is provided by the exporterhelper module and the gRPC client library; none of it is internal to this component.

• Counter of successes and failures to send a value, per signal type
• Counter of successes and failures to enqueue a value, per signal type
• Current queue size and capacity, in batches
• gRPC: request size, request duration, response size, number of messages in request/response

Proposals for new metrics

I believe all the proposals from receiver/otlp are also applicable here.

12. Counter of retries in the retry_exporter
13. Counter of requests that timeout after retries
14. Queue size in units and in bytes
15. Network roundtrip latency of export (gRPC request duration may be equivalent?)

Additional notes

While reviewing the code, I noticed that the "queue capacity" metric, unlike the "queue size" metric, does not set a "signal type" attribute. I believe this should only be an issue in the (unsupported?) edge case where an exporter hardcodes different queue configs for each data type instead of passing the user-provided config to exporterhelper.NewTraces/Logs/Metrics.

I also saw mention of observing "end-to-end latency from receiver input to exporter output". Not sure which component this would belong in, and it might be tricky to implement.

exporter/otlphttp

This is the same for as exporter/otlp, except the lower-level request metrics come from the HTTP client:

• HTTP: request size, request duration, response size, active request count, connection count, connection duration.

Proposals and notes are the same as above.

[jade-guiton-dd]

Now that observability requirements for stable components have been formally defined, we can assess more precisely which of the proposals above will need to be implemented before stabilization.

Unlike for the previous review, I will split this up between the issues for the different OTLP components.

receiver/otlp

2. How much data the component outputs.

✅ This should be covered by pipeline instrumentation, once implemented.

5. Other possible discrepancies between input and output, if any.

✅ The receiver does not willfully drop items, create them, or hold them, so this is not applicable.

6. Processing performance.

⚠️ Currently, the span emitted by receiverhelper only encompasses downstream processing, so the different in timing between it and the span emitted by otelgrpc/otelhttp can be used to determine the latency. However, this is an implementation detail, and should probably not be relied on.

⚠️ Moreover, receiverhelper does not currently support profiles. This means latency is not observable for them, which would be a blocker if profiles were a stable signal.

➡️ Possible solution: If pipeline instrumentation is implemented in a way that generates spans for each Consume operation (see issue #11743 for discussion), then the difference between that and the otelgrpc/otelhttp span may be used to unambiguously measure latency. In that case, no additional telemetry would be necessary from the receiver or receiverhelper.

➡️ An alternate solution would be to modify the spans emitted by receiverhelper to make sure that users can differentiate between spans covering the entire component operation and spans covering downstream processing. This could be done by modifying its API to create two spans, for example.

All internal telemetry emitted by a component should have attributes identifying the specific component instance that it originates from. This should follow the same conventions as the pipeline universal telemetry.

In the case of "singleton" receivers like the OTLP receiver, the component ID (eg. otlp or otlp/foo) is enough to identify the instance.

⚠️ The receiver attribute on receiverhelper metrics and the name of receiverhelper spans allows users to identify the source instance. It is also possible for spans and metrics generated by otelhttp, by comparing the net.host.port attribute with the config. However, this is not uniform and does not follow pipeline telemetry conventions.

❌ otelgrpc spans and metrics do not give any port information, which prevents identifying the source component (in cases where no receiverhelper span is emitted). This is problematic when there are multiple gRPC endpoints configured.

➡️ To address the most pressing issue, an issue should be filed on otelgrpc to implement the server.address and server.port attributes defined in the semantic conventions.

➡️ My long-term recommendation would be to modify receiverhelper to emit attributes according to the pipeline telemetry conventions, and to inject said attributes into otelgrpc and otelhttp as well. For otelhttp, this can be done with ContextWithLabeler for metrics and WithSpanOptions for spans; for otelgrpc, with WithMetricAttributes and WithSpanAttributes.

HTTP path

1. How much data the component receives.

✅ The number of HTTP requests can be observed using the count of the http.server.request.duration histogram, or the number of traces created by otelhttp. The size of the requests can also be observed using http.server.request_size.

3. How much data is dropped because of errors.
4. Details for error conditions.

There are 3 error modes for HTTP requests:

• Failure to read request: (data dropped because of internal error)
  (Possible reasons: invalid method, invalid content type, failure to read body, failure to unmarshal body)
  HTTP 405/415/400 returned (observable using otelhttp span and metric attributes)
  Success span emitted by otelhttp
• Downstream error: (data dropped because of downstream error)
  HTTP 500 returned
  Error spans emitted by otelhttp and receiverhelper; the latter contains the error message
  otelcol_receiver_refused_xxxx metric emitted, and also recorded by pipeline instrumentation
• Failure to marshal response: (no data dropped, indicative of a bug)
  HTTP 500 returned
  Error span emitted by otelhttp
  Success span emitted by receiverhelper

⚠️ From this breakdown, we can see that it is technically possible to differentiate all three cases, which allows counting failed requests and distinguishing downstream errors from internal errors. However, this requires correlation between multiple data points. Moreover, in case 1, the error details can only be deduced from the status code.

⚠️ For profiles, no spans are currently emitted by receiverhelper, and the downstream error messages are not recorder. This would be a blocker if profiles were not currently experimental.

➡️ A possible improvement would be:

• Adding an otelcol_receiver_http_requests counter metric, with an outcome attribute similar to that set by pipeline instrumentation. If my amendment of the RFC (Amend Pipeline Component Telemetry RFC to add a "rejected" outcome #11956) lands, this would have three values (success, failure, refused), to differentiate between success, internal errors, and downstream errors.
• Log the error messages for cases 1 and 3, either as a log, or as a span event on the otelhttp span.

I am unsure if failures to marshal the response should be a success or a failure, as no data is dropped. This case should be fairly anecdotal however, especially since the response message does not typically contain any data.

➡️ A more general solution may be to update the receiverhelper API, in a way that would allow us to output errors as failed spans all 3 cases, and to automatically output a metric with the appropriate outcome attribute. This is similar to, and may be combined with, the second suggestion in the "Processing performance" section.

gRPC path

1. How much data the component receives.

✅ rpc.server.duration, rpc.server.request.size, and the traces created by otelgrpc, can be used in the same way as their HTTP equivalents.

3. How much data is dropped because of errors.
4. Details for error conditions.

There are 4 error modes for gRPC requests:

• Basic protocol errors (wrong protocol, wrong path or content-type):
  An error is returned to the client, but no telemetry is emitted.
• Errors in payload unmarshaling, potentially other internal gRPC errors:
  Error span emitted by otelgrpc
• Early connection shutdown:
  Success span emitted by otelgrpc (?)
  info level log with no span id emitted
• Downstream error:
  Error spans emitted by otelgrpc and receiverhelper; both contain the error message
  otelcol_receiver_refused_xxxx metric emitted, and also recorded by pipeline instrumentation

⚠️ Some protocol and connection-related errors are not properly observable. We could consider this data loss, but we may also consider monitoring these to be the responsability of the client?

➡️ If we decide we want to monitor more types of errors on the Collector side, my recommendation would be to open an issue with otelgrpc to check whether this information is exposed by the "stats" system the gRPC library provides.

[jade-guiton-dd]

To sum up, it looks like there are 2 strict blockers for stabilization according to the requirements we defined:

• otelgrpc adding attributes allowing users to identify the source receiver (tracked in [otelgrpc] Emit attributes identifying host endpoint go-contrib#6608)
• receiverhelper has been modified to use the same component-identifying attributes as the RFC: [receiverhelper] Implement new component-identifying attributes #12117
  → This should probably wait until a generic way to gather said attributes has been implemented: Add singleton flags to factories and standardize attributes #12057 should be a good first step

Update: The latter issue was resolved by #12617, which injects component attributes as defined by the RFC everywhere. Because of this, the first issue is no longer a blocker, as the source receiver can be identified from said attributes.

However, if we want to avoid adding new telemetry or immediately deprecating telemetry post-1.0, I would suggest waiting until:

• a conclusion has been reached in [Universal Component Telemetry] Revisit tracing #11743 on whether anything should be done with spans
• receiverhelper has been further modified based on the above suggestions:
  • outputting a metric tracking the outcome of requests: [receiverhelper] Add metric for requests that failed to be received #12207
  • clarifying the semantics of the emitted spans (waiting on [Universal Component Telemetry] Revisit tracing #11743), potentially emitting more
• otelgrpc updates to a more recent semantic convention? (currently using v1.17.0, which is 2 years old)

Note that the above is only true for traces/metrics/logs. The fact that receiverhelper does not currently support profiles is an additional blocker for stabilization of the component for profiles.