otelcol: use sync.Once in Shutdown

[axw]

Description

Make sure that Shutdown closes the shutdown channel at most once, but don't use the collector state to determine this. That's because the state may be "closing" during a config reload.

Link to tracking issue

Fixes #13740

Testing

Applied diff in the linked issue, does not trigger the bug anymore.

Documentation

N/A

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.40%. Comparing base (e8135a8) to head (9cb981f).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #13741      +/-   ##
==========================================
- Coverage   91.45%   91.40%   -0.06%     
==========================================
  Files         640      640              
  Lines       42668    42664       -4     
==========================================
- Hits        39023    38996      -27     
- Misses       2828     2845      +17     
- Partials      817      823       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[dmathieu]

Have you tried reproducing the issue in a unit test?

[axw]

@dmathieu only briefly. I can take another look tomorrow - basically we'd need to block the internal service shutdown on reload, and check that a Shutdown call isn't ignored.

[axw]

@dmathieu done - the test now reliably triggers the bug on main

[bogdandrutu]

Should we have a "reloading" state instead and shutdown even in that?

[axw]

@bogdandrutu IMO yes, but not as a solution to this particular problem

[bogdandrutu]

I think also we need to make a decision about Run after Shutdown and make sure clearly implemented in the code.