Profile: Clarification Needed for `SampleType`, `Value` and `TimestampsUnixNano`

[YifanYang6]

Component(s)

profile

Describe the issue you're reporting

Hello,

I've encountered a potential inconsistency between the proto definition for profile data and the actual data observed during debugging. Specifically, I'm concerned with the behavior of the Value and TimestampsUnixNano fields within the Sample structure.

According to the proto definition https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/profiles/v1development/profiles.proto#L375-L396 it states:

"All samples must have the same number of values, the same as the length of Profile.sample_type"

However, my debugging output shows the following:

Each Value entry appears to correspond to a specific Timestamp, rather than being determined by the Profile.sample_type length.
Samples within the same profile do not consistently have the same number of Value or Timestamp entries.
Here's an example of my debugging output:

Profile #1
    Profile ID     : c2d01967aa7b578f1b70e3c35fd6e379
    SampleType #%d : %!s(int=0)
    TypeStringIndex: events
    UnitStringIndex: nanoseconds
    Start time     : 2025-03-05 04:56:59.894095226 +0000 UTC
    Duration       : 1970-01-01 00:00:01.756312228 +0000 UTC

    Sample #0
        Location start: 0
        Location length: 6
        Value: [903976986 839861575 883929035 463890732 668006176]
        Attributes:
             -> container.id: /
             -> thread.name: kworker/0:0
             -> process.pid: %!s(int64=2548176)
        Timestamp: 1741150617770086999
        Timestamp: 1741150618610126241
        Timestamp: 1741150619494089058
        Timestamp: 1741150619958085800
        Timestamp: 1741150620626113008
    Sample #1
        Location start: 6
        Location length: 6
        Value: [20025858 1007966728 7908106]
        Attributes:
             -> container.id: /
             -> thread.name: kworker/12:2
             -> process.pid: %!s(int64=4041331)
        Timestamp: 1741150618074111301
        Timestamp: 1741150619082111579
        Timestamp: 1741150619090073827

Based on these observations, I have the following questions:

1. What is the intended relationship between Value, TimestampsUnixNano, and Profile.sample_type?
2. Why do individual samples contain multiple Value and Timestamp entries? What does this represent?
3. Is it a strict requirement that all samples within a profile share the same sample_type? I've noticed relevant checks in the Elasticsearch Exporter (https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/elasticsearchexporter/internal/serializer/otelserializer/serializeprofiles/transform.go#L46).

I'm new in OpenTelemetry and profiling, and currently want to implement a persistent storage for profile data. Now I'm working on open-telemetry/opentelemetry-collector-contrib#38077. I would greatly appreciate if anyone could clarify these points!

[github-actions]

Pinging code owners:

• pdata/pprofile: @mx-psi @dmathieu

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dmathieu]

Note that this code is auto-generated, as is the doc.
The source of reference is here: https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/profiles/v1development/profiles.proto

The collector does not generate any profiling data. It gets it from the pipeline (OTLP, or the ebpf-profiling receiver), and passes it down. So this looks like a question for @open-telemetry/profiling-approvers.
How are you generating those profiles? If you're using the ebpf profiler, that repository may be a better place for this question.

[YifanYang6]

Note that this code is auto-generated, as is the doc. The source of reference is here: https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/profiles/v1development/profiles.proto

The collector does not generate any profiling data. It gets it from the pipeline (OTLP, or the ebpf-profiling receiver), and passes it down. So this looks like a question for @open-telemetry/profiling-approvers. How are you generating those profiles? If you're using the ebpf profiler, that repository may be a better place for this question.

Thank you for your response.
Yes the data is generated with ebpf profiler. current pipeline is ebpf-profiler -> otel-collector. And collector config is

extensions:
  pprof:
    endpoint: 0.0.0.0:1778

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4333
      http:
        endpoint: 0.0.0.0:4444

processors:
  batch:

exporters:
  debug:
    verbosity: detailed

service:

  pipelines:
    profiles:
      receivers: [otlp]
      exporters: [debug]

  extensions: [pprof]

To enhance the debug output, I'm using a custom build with some patches for debug exporter from LGU-SE-Internal/opentelemetry-collector@f3865c3, as the standard debug exporter lacked some crucial information.

I understand I go to the wrong repo, so it's better to open the issue in https://github.com/open-telemetry/opentelemetry-ebpf-profiler right?

Thank you for your guidance!

[YifanYang6]

Thank @mx-psi for transfer the issue

I am currently unable to determine whether the observed data discrepancies are due to necessary adjustments in the profiles.proto definition, or if they result from the ebpf-profiler's OTLP exporter not fully adhering to the proto definition.
I have observed that the Elasticsearch Exporter's implementation
https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/2d3edff0ead36b16f032e9bcd87f9b7b95252129/exporter/elasticsearchexporter/internal/serializer/otelserializer/serializeprofiles/transform.go#L115-L126
associates timestamps with values.

[dmathieu]

Note that all of profiling is currently in flux, as it's under heavy development.
There can, has been and will be breaking changes in the proto definitions.

[YifanYang6]

Note that all of profiling is currently in flux, as it's under heavy development. There can, has been and will be breaking changes in the proto definitions.

I see, it's normal to have breaking changes in the development stage.
Appreciate the hard work from the team so far; it’s been quite a journey! also I'm glad to make some contribution in any way I can

[felixge]

Note that all of profiling is currently in flux, as it's under heavy development. There can, has been and will be breaking changes in the proto definitions.

I see, it's normal to have breaking changes in the development stage. Appreciate the hard work from the team so far; it’s been quite a journey! also I'm glad to make some contribution in any way I can

👋 Hey, I'm also part of the profiling SIG. If you'd like to get more involved, feel free to join us on slack or join one of our meetings (every 2nd thursday). See open-telemetry/community#sig-profiles for more details. I'd also be happy to chat to explore specific opportunities for contributions.

[felixge]

Thanks for reaching out! I’ve already joined Slack and would be happy to attend the meetings as well. Would it be possible to discuss these topics in the upcoming meeting on March 6? Just to confirm, I’m guessing it’s in about 13 hours from now, right?

Yes, we can definitely discuss it. I'll add an agenda item to our meeting notes. Feel free to add stuff there in the future yourself.

[YifanYang6]

Thanks for reaching out! I’ve already joined Slack and would be happy to attend the meetings as well. Would it be possible to discuss these topics in the upcoming meeting on March 6? Just to confirm, I’m guessing it’s in about 13 hours from now, right?

Yes, we can definitely discuss it. I'll add an agenda item to our meeting notes. Feel free to add stuff there in the future yourself.

Thanks for adding it! The last message was actually from me using a personal account by mistake😅—I’ve deleted it to avoid confusion. Looking forward to discussing the topics!

[felixge]

@florianl and I had a chance to discuss this at KubeCon in person and would like to propose the following:

• Every Profile message should have just a single Profile.sample_type.
• At least one of Sample.values or Sample.timestamps_unix_nano must be non-empty. The other field can either be empty, or have the same number of elements, in which case the elements at the same index represent a (timestamp, value) tuple.

Note: The values field is currently named value, but it should be renamed to values.

Examples

5 CPU Samples with Timestamps at the same Stack Trace

Profile{
  sample_type: {type: 'cpu', unit: 'count'}
  sample: {
    values: []
    timestamps_unix_nano: [1, 2, 3, 4, 5]
    // stack trace reference, attributes, etc.
  }
}

5 CPU Samples without Timestamps at the same Stack Trace

Profile{
  sample_type: {type: 'cpu', unit: 'count'}
  sample{
    values: [5]
    timestamps_unix_nano: []
    // stack trace reference, attributes, etc.
  }
}

5 Off-CPU Samples with Timestamps at the same Stack Trace

Profile{
  sample_type: {type: 'off_cpu', unit: 'nanoseconds'}
  sample: {
    values: [10, 30, 20, 90, 110]
    timestamps_unix_nano: [1, 2, 3, 4, 5]
    // stack trace reference, attributes, etc.
  }
}

5 Off-CPU Samples without Timestamps at the same Stack Trace

Essentially a histogram. There might be use cases where people don't need the individual timestamps and just care about the distribution off Off-CPU waits.

Profile{
  sample_type: {type: 'off_cpu', unit: 'nanoseconds'}
  sample: {
    values: [10, 30, 20, 90, 110]
    timestamps_unix_nano: []
    // stack trace reference, attributes, etc.
  }
}

Go Memory Profile

Go memory profiles will become a little bit more verbose compared to their current pprof encoding, in which the profile has 4 sample types and all the values for the same stack trace are encoded together.

We think this might cause memory profiles to become ~2x larger, but since CPU profiles are usually larger, we think this is not going to matter too much in practice and allow for an overall simpler design when it comes to supporting timestamps for the other profile types.

ScopeProfiles{
  profile: {
    sample_type: {type: 'allocated_objects', unit: 'count'}
    sample: {
      values: [90]
      timestamps_unix_nano: []
      // stack trace reference, attributes, etc.
    }
  },
  profile: {
    sample_type: {type: 'allocated_space', unit: 'bytes'}
    sample: {
      values: [11520]
      timestamps_unix_nano: []
      // stack trace reference, attributes, etc.
    }
  }
  profile: {
    sample_type: {type: 'inuse_objects', unit: 'count'}
    sample: {
      values: [30]
      timestamps_unix_nano: []
      // stack trace reference, attributes, etc.
    }
  },
  profile: {
    sample_type: {type: 'inuse_space', unit: 'bytes'}
    sample: {
      values: [3840]
      timestamps_unix_nano: []
      // stack trace reference, attributes, etc.
    }
  }
}

[rockdaboot]

👍 Thanks for this, makes sense to me.

@felixge Just a request for some little changes in the comment above

1. The part "..., or have 0 elements" is a bit ambiguous to me. Isn't it that at least one of the two fields must not be empty and then the other field must have either 0 elements or the same number of elements!?
2. Sometimes you refer to timestamps, but I guess you meant timestamps_unix_nano?
3. Can it say value instead of values, as defined in the protobuf definition?
4. In the second example there is no mention of timestamps_unix_nano.