SDK exporter health metrics: Possibly enhance with duration

[JonasKunz]

#1631 added SDK health metrics for span-related processing. A discussion that came up towards the end of this PR is whether to add a duration metric for exporters in addition to or as replacement for otel.sdk.exporter.span.exported.count.
See this comment and this comment.

The goal of this issue is to evaluate that addition and maybe also consider exporter-request metrics for duration, size and errors.

[JonasKunz]

To continue the discussion here, the content of the last comment:

At the same time I can't really see the general importance / usefulness of having the exporter durations: It feels like more of a nice to have. What conclusions does this metric allow you to draw? Do you have concrete examples?

Debugging connectivity with my backend - network issues, throttling, slow backend response, retries, retry backoff interval optimizations.
As @lzchen mentioned, absolute majority of OTel SDKs suppress all instrumentation around exporter calls, so you'd not have protocol-level spans and/or metrics. So a metric like this would be the only way to know how good your connectivity is.

Counts are good, but they won't tell you that your P99 is 10 sec after all tries because your backoff interval is wrong. You'd just see less of them and will have no idea.

I don't think it's nice to have.
Given it's experimental, we can go ahead with just counts and potentially deprecating them when/if histograms are added.

As a cost mitigation strategy, we can always use a small amount of buckets by default and users can always reconfigure them if they need less/more.

Here are my thoughts:

Debugging connectivity with my backend - network issues, throttling, slow backend response, retries, retry backoff interval optimizations.

I think for those it is better to have dedicated otel.sdk.exporter.request.* metrics for request-based exporters. We can lend the overall design from the existing HTTP metrics here. I think that should also be applicable to other types of network requests, too:

• otel.sdk.exporter.request.duration
• otel.sdk.exporter.request.body.size

These metrics would be applicable to all kinds of exporters (span, metric, traces) and should cover the above use-cases except for retries.

IIRC retries was one of the motivations to go for a otel.sdk.exporter.span.duration metric: It would cover the time a span takes from being submitted to the exporter until it is either exported successfully or is dropped as failed, including potential retries.

My question here would be how to correctly track this, considering exporter requests can have partial success semantics. E.g. imagine the following sequence of events:

• A batch of spans is submitted to the exporter
• The request fails for a network reason
• In the meantime, a second batch of spans is submitted for export
• The second request is issued, containing the spans from both batches
• The request finishes with partial success, returning that 5 spans were rejected

In this scenario, it is not possible to know if those rejected 5 spans were from the first or the second batch, which therefore have different durations. This is however needed to be known to correctly report the duration metric with the correct error.type tag.

This is why I think at the moment it is not possible, to correctly define a otel.sdk.exporter.span.duration metric.

Nonetheless, if we find a way of defining otel.sdk.exporter.span.duration correctly, I still think such histogram metrics (including otel.sdk.exporter.request.duration and otel.sdk.exporter.request.size) should be opt-in:

As @lzchen mentioned, absolute majority of OTel SDKs suppress all instrumentation around exporter calls, so you'd not have protocol-level spans and/or metrics. So a metric like this would be the only way to know how good your connectivity is.

You still have the traditional escape hatch of logs: The SDK logs will tell you in detail what went wrong. So with just the existing count metrics you'd have to jump into those to analyze the problem.

I previously raised the concern of the much higher storage cost of histograms:

As a cost mitigation strategy, we can always use a small amount of buckets by default and users can always reconfigure them if they need less/more.

I think this would lead to users building bad dashboards by accident and it being difficult / cumbersome to reconfigure the bucket boundaries.

Why would users build bad dashboards? Well, they would see metrics of type histogram and visualize them accordingly (e.g. extract a P99). They would have to dig into the data / specs to actually see that the the bucket sizes are not suited for this visualizations by default.
Secondly, reconfiguring the bucket boundaries for a single, existing metric can be cumbersome, thinking of e.g. autoinstrumentation commonly used in java. I don't think you could get a way doing that without requiring the user to write code. Or we'd have to provide a config option to enable better bucket boundaries, which also seems a bit weird and specific to me.

That's why I think it would be better to have the .count metrics enabled by default and have additional histogram metrics opt-in. This way, you get data-loss monitoring out of the box for a low storage cost. If you don't want to have to jump into the SDK logs if those show data loss, you'd have to opt-in into histogram metrics, which will help you locate the problem better at a higher storage cost.

The only downside is that then you waste a little space due to redundancy. However, considering that if you have e.g. 20 histogram buckets, the additional redundant count is a waste of just 5% additional time-series (and therefore storage), which IMO is very acceptable. The user opted into more detailed metrics, so he's okay with using more space.

So @dashpole @lzchen @lmolkova I'd be happy to hear your thoughts here:

• Do you agree or have a different opinion, whether histogram metrics should be an opt-in in addition to .count metrics?
• Thoughts about my concerns regarding the otel.sdk.exporter.span.duration metric?

[dashpole]

I'm still planning to play around with combining the existing count metric and the http/gRPC metrics to see if those are enough.

[lzchen]

IIRC retries was one of the motivations to go for a otel.sdk.exporter.span.duration metric: It would cover the time a span takes from being submitted to the exporter until it is either exported successfully or is dropped as failed, including potential retries.

Could you provide an example of a retry scenario in which otel.sdk.exporter.span.duration can track this but otel.sdk.exporter.request.* could not? I don't fully understand the motivation here.

Secondly, I would think that whatever metric(s) we choose to define should be able to be applied to all traces, metrics and log exporters, which is why we would want something like otel.sdk.exporter.request.* or otel.sdk.exporter.<span,log,metric>.duration would make sense.

In this scenario, it is not possible to know if those rejected 5 spans were from the first or the second batch, which therefore have different durations. This is however needed to be known to correctly report the duration metric with the correct error.type tag.

1. I would think the first request records a otel.sdk.exporter.span.duration metric with error.type of whatever the network response was.
2. The second request would record a otel.sdk.exporter.span.duration metric with success or error.type 206 (not sure if we want to treat a batch as success of error if partial success). It would also then increment otel.sdk.exporter.<span/log/metric>.exported.count per each error.type or success.

I believe duration metrics should represent the batch request instead of on per item level. Having the duration metric which is supposed to be on the batch level contain an error.type tag that's related to an item in the batch doesn't make much sense to me.

Apologies if I am not understanding properly the scenario.

[lzchen]

That's why I think it would be better to have the .count metrics enabled by default and have additional histogram metrics opt-in.

I think you raise good points about histogram. Makes sense to me for being opt-in.

[JonasKunz]

IIRC retries was one of the motivations to go for a otel.sdk.exporter.span.duration metric: It would cover the time a span takes from being submitted to the exporter until it is either exported successfully or is dropped as failed, including potential retries.
Could you provide an example of a retry scenario in which otel.sdk.exporter.span.duration can track this but otel.sdk.exporter.request.* could not? I don't fully understand the motivation here.

Well, my suggested basic metrics are per-request:

• otel.sdk.exporter.request.duration
• otel.sdk.exporter.request.body.size

Each request, independently of whether it contains some payload which is being retried or the content is fully new, submits a datapoint to those metrics. There are no additional labels for marking retries I could think of which make sense: Like I explained in my previous scenario, the payload of a request can be a mix of retried data and new data.
For tracking retries, an additional metric would be needed ,e.g.:

Histogram otel.sdk.exporter.request.retry_count: Per request, records the "age" of the oldest bit of the payload. "age" hereby means how often the data (e.g. a span) has been tried to be sent.
So if a request contains a span which is tried for the third time (=second retry) and a new span, the value to submit to the metric would be 2.

Secondly, I would think that whatever metric(s) we choose to define should be able to be applied to all traces, metrics and log exporters, which is why we would want something like otel.sdk.exporter.request.* or otel.sdk.exporter.<span,log,metric>.duration would make sense.

Agree, that's why I proposed otel.sdk.exporter.request.* and not otel.sdk.exporter.span/metric/log.request.*.

In this scenario, it is not possible to know if those rejected 5 spans were from the first or the second batch, which therefore have different durations. This is however needed to be known to correctly report the duration metric with the correct error.type tag.

1. I would think the first request records a otel.sdk.exporter.span.duration metric with error.type of whatever the network response was.
2. The second request would record a otel.sdk.exporter.span.duration metric with success or error.type 206 (not sure if we want to treat a batch as success of error if partial success). It would also then increment otel.sdk.exporter.<span/log/metric>.exported.count per each error.type or success.

I believe duration metrics should represent the batch request instead of on per item level. Having the duration metric which is supposed to be on the batch level contain an error.type tag that's related to an item in the batch doesn't make much sense to me.

The existing definition of otel.sdk.exporter.span.exported.count is intentionally on per-item level to detect data loss: We want to be sure that the count is precisely the same as the number of ended spans. It would be very confusing to have a otel.sdk.exporter.span.duration on the same namespace which is on batch level. You should then name it accordingly otel.sdk.exporter.span.batch.duration. However, that still is not trackable based on my initial scenario given:
The second request in fact contains two batches: the retried one and the new one. Now with a partial success, it is not possible to know if one of the batches was a full success or both are only partial successes.

[lzchen]

Each request, independently of whether it contains some payload which is being retried or the content is fully new, submits a datapoint to those metrics. There are no additional labels for marking retries I could think of which make sense: Like I explained in my previous scenario, the payload of a request can be a mix of retried data and new data.
For tracking retries, an additional metric would be needed ,e.g.:
Histogram otel.sdk.exporter.request.retry_count: Per request, records the "age" of the oldest bit of the payload. "age" hereby means how often the data (e.g. a span) has been tried to be sent.
So if a request contains a span which is tried for the third time (=second retry) and a new span, the value to submit to the metric would be 2.

Yup I agree with this. For our initial scenario, we only need otel.sdk.exporter.request.duration for now as we aren't interested on whether the spans in the request are original or are from a retry attempt.

otel.sdk.exporter.span.batch.duration ... However, that still is not trackable based on my initial scenario given:

I also agree that this is not trackable since span is on an item level. I think our ask is only for on the request level with otel.sdk.exporter.request.duration.

If we agree that otel.sdk.exporter.request.duration would be a good addition, I believe the only (possible) point of contention is how we represent partial successes. Since this metric would be dedicated to counting batch level metrics, I don't think it's required to drill into which items were success/failure/retry for this metric at least. Separately, if we are interested in item level, it might be up to implementations on how they handle recording item level results with otel.sdk.exporter.span.exported.count.

[JonasKunz]

if we are interested in item level, it might be up to implementations on how they handle recording item level results with otel.sdk.exporter.span.exported.count

That is not permitted by the wording in the current spec, as this metric is recorded when a span has either succeeded or failed, while a span being retried is still in flight.

Other than that I think we are on the same page.

I believe the only (possible) point of contention is how we represent partial successes

I would suggest to handle it similarly as for the count metrics: Only for fully successful requests, error.type is unset. For everything else (failures and partial success), error.type needs to be set to some meaningful value.

[JonasKunz]

For future reference: In the last SIG-meeting (2025/04/07) we came to the conclusion that it is not necessary to have histogram metrics always opt-in:

• To limit storage size, recommended metrics should use only very little (e.g. one) buckets. The current inconvencience for users to reconfigure these buckets will greatly improve with file-based configuration.
• A histogram type misleading users into building bad dashboards was decided to be a non-issue: When there is just one bucket it will be obvious