Sanitize OTLP profiles

[kolesnikovae]

We should ensure that profiles which don't conform to the specification are handled gracefully, without causing data corruption or service failure.

Even though it's still an experimental feature in development, we should prioritize safety and a good user experience.

[korniltsev]

Just braindumping here: maybe in a future we could add a fuzzer to surface at least some of invalid profile panics.

[fandreuz]

I could work on this, I wanted to have a look at the problem I reported here anyway.

[kolesnikovae]

Just braindumping here: maybe in a future we could add a fuzzer to surface at least some of invalid profile panics.

Most definitely we need a fuzzer in addition to standard unit tests.

I could work on this, I wanted to have a look at the problem I reported #4212 (comment) anyway.

Thank you – that would be incredibly helpful.

I propose that we drop profiles that contain malformed elements, rather than trying to preserve valid parts. Currently, we do the opposite when processing pprof.

Also, please keep in mind that pprof profiles are normalized afterward – we do some extra sanitization here: https://github.com/grafana/pyroscope/blob/main/pkg/pprof/pprof.go#L357-L356; I'm happy to answer any questions

[fandreuz]

Hi @kolesnikovae, thanks a lot for the inputs. I appended two commits on top of #4212 to handle gracefully malformed OTLP profiles, and to add additional testing around those cases. Let me know what you think about the approach I took there.