Update receiverhelper for requests that failed to be received

[gizas]

Description

This is an ongoing work to add :

• In the endOp function of ObsReport, errors are classified as either:
  • Downstream errors (refused) - when internal.IsDownstreamError(err) returns true
  • Internal errors - when the error is not a downstream error
• New metrics otelcol_receiver_internal_errors_spans, otelcol_receiver_internal_errors_metric_points and otelcol_receiver_internal_errors_log_records in case of inetrnal errors returned above
• Support of otelcol_receiver_requests metric that will determines the outcome of the request:
  • "success" if there's no error
  • "refused" if there's a downstream error
  • "failure" if there's an internal error

Link to tracking issue

Fixes #12207

Testing

Only added the integration tests.

Documentation

See updated receiver/receiverhelper/documentation.md

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: gizas / name: Andrew Gizas (0bb9732, 250f1ba, 140fc8f, 81c6f86, 9294a16, 6f013a7, 1c151b4, b76a51d, 79cc00c, cb2796a, d4c452a, 8a2116c, ca361bb, 3decea5, 61e3468, c3439a8, ed48254, 0194ac5, d7f9533, 3ab59d3, 75e3e7f, 1803e33, 85bcabb, 515e4d5, 9b7c3f5, fcc1943, d729b2c, 5b0be26, 2928c47, a6f7dc7, d9eebb1, fdb2762, 702d48a, 535d2ba, 843ef40, 08fb90d, 45ab725, 165b076, d20d19d, 579e532, cd2bb19, 666c5b2, ad79665, 6ee5ec8, 44b193e, dd3df23, 0a38936, 0018f77, d347bd1, b3b8b87, 8a4a26c, 0c4b977, b926b14, 2c6b080, e475745, a92c7a6, 36acb84, 4846daf, a6db9e1, c82a466, cd6e640, 53992b5, 143b04c, b9e788d, 29e209d, 1053f75, b77091f, 25ea91a, eb93ce5, 6583a43, 0019011, 156f422, 9a6cf12, 24e4281, 73fd278, f0297ff, 83b9462, 89d7bc4, 7fc9765, 52b302d, b9543ac, 3267177, de8ed1b, ec402d5, a1c0105, 0d3e4b1, 89c07dd, 041e3f6, 1f2a196, a8a0b2d, fac2c12, 4fd8bc6, 5637ee6, 8199a4b, 44c898c, 638370f, 909a081, 3601749, 7f5d19f)

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[jade-guiton-dd]

@gizas Are you still working on this? Or should it be marked as ready for review?

[gizas]

@jade-guiton-dd I will try to fix the linting errors and will open it as ready for review. thanks for the ping

[gizas]

@jade-guiton-dd I think I have made some progress. I have made the choice to add a new flag additional to existing telemetry.newPipelineTelemetry
So if someone now wants to test this should run:

otelcol --config=config.yaml --feature-gates=+telemetry.newPipelineTelemetry,+telemetry.distinguishDownstreamErrors

Screenshot with some manual tests

I have even built and run some manual tests and I can see e2e the metrics appear:

Let me know wdyt

[gizas]

Hm, it seems pretty weird that your PR would affect the coverage of tests in the exporterhelper. Are you sure it's related to your changes, and isn't due to a change that was made on main?

No are not related to my changes. But are indirect refernces. See the links above. Ok I will also remove them. I agree will be cleaner

[jade-guiton-dd]

Looking pretty good, although there are still a few issues to fix. The go.mod required versions are also still wrong.

[jade-guiton-dd]

I'm surprised that test fails considering the feature gate should be disabled by default 🤔

[jade-guiton-dd]

Looking at the contrib-tests currently running in CI, I don't see any failures for receiver/awsxrayreceiver. I'm wondering if you weren't testing against an older version of your code, with the incorrect logic in obsreport?

[gizas]

Looking at the contrib-tests currently running in CI, I don't see any failures for receiver/awsxrayreceiver. I'm wondering if you weren't testing against an older version of your code, with the incorrect logic in obsreport?

Indeed @jade-guiton-dd our comments crossed. I wrote it the comment for the contrib-tests before seeing yours to update the logic. So later I did the updates. (I saw some errors today but unrelated to our changes). I have fixed the lint errors and retriggered the CI. Let's see now

[gizas]

@jade-guiton-dd do you think are we ready to merge? anything else missing?

[jade-guiton-dd]

I think we're about done, I just have a few more suggestions regarding documentation. Thank you for your patience with all these back-and-forths.