Add pages for collector components

[jaydeluca]

I am experimenting with automation that scrapes collector metadata and generates these pages. Github action jobs will run nightly that look for changes, and will regenerate the pages and submit PRs accordingly.

This is a starting point, and likely to evolve as we expand the data we include for each component.

cc @tiffany76

---

PREVIEW: https://deploy-preview-8001--opentelemetry.netlify.app/docs/collector/components/

[jaydeluca]

i wasnt sure whether it made sense to include the x components?

[mx-psi]

I think we can exclude those

[dmathieu]

This is nice. But looking at it, it could be read that those are the only components anyone can use, which isn't true (if you build your own distribution, you can use any component, including private ones).

Speaking of distributions, there are 5 different official collector distributions. How about adding which of them each component is included into?

[tiffany76]

Thanks for taking a look at this POC, @dmathieu. We're just getting started and can use expert input!

We plan to update the language on the pages and will not only call out that users can use other components, but also cross-reference the registry, which includes many that are not part of core/contrib.

We're starting with a light lift for this POC while we investigate exactly what should be included and how it should be displayed. I'll add the otelcol distribution to our list of what should be included.

Thanks again!

[mx-psi]

This looks great!

Some ideas (not necessarily for this first version, just writing them down somewhere):

1. I wonder if it would make sense to style/order components differently depending on their stability level (since the stability level can vary per signal I guess the following can be done by taking the max stability level).
   For example, we could fade out in development components and strike out those that are unmaintained, and/or order components based on their max stability level (so that first you see stable components, then beta components, then alpha and lastly development). We could even separate alpha components into a collapsed box to signal that they are more experimental.
2. We are missing profiling support from the list! This is an experimental signal so maybe can be excluded
3. I think it would be great to have links to https://opentelemetry.io/docs/collector/configuration/#receivers and similar somewhere in these pages
4. It would also make sense to me to link to https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md somewhere

[mx-psi]

I think we can exclude those

[chalin]

A few broad questions and comments:

• Have you seen https://opentelemetry.io/status/#collector? The added page content seems to be mostly statuses for components. Might it make sense to put these pages (or at least their status content), under https://opentelemetry.io/status/#collector or a new subsection of https://opentelemetry.io/status/?
• Re. the automation: under which repo will the GH actions run (and where will be the associated scripts, if any)?
• There is https://opentelemetry.io/docs/collector/distributions/. Is there an overlap with what is being proposed here as content? If so, will it lead to confusion for readers?

[chalin]

OTel defines official levels of maturity in the spec(s) and elsewhere, such as https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md (as you probably know :). Could we link to that page as the main source of truth?

[tiffany76]

@mx-psi, thank you for the review!

I wonder if it would make sense to style/order components differently depending on their stability level (since the stability level can vary per signal I guess the following can be done by taking the max stability level).

That's an interesting idea. We should consider how to offer filtered/sorted views when we iterate on how the data is presented, especially if we move away from tables. 👍

We are missing profiling support from the list! This is an experimental signal so maybe can be excluded

Noted!

I think it would be great to have links to https://opentelemetry.io/docs/collector/configuration/#receivers and similar somewhere in these pages

I was thinking these pages will merge with that content to actually become the /receivers, /processors, /exporters, etc. pages in the new IA. So the corresponding content on that monolithic configuration page will be broken out into separate pages, and each page will have the OTel-maintained components listed (right now, in a table). WDYT?

It would also make sense to me to link to https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md somewhere

Noted! I think Patrice had the same idea. 👍

[mx-psi]

I was thinking these pages will merge with that content to actually become the /receivers, /processors, /exporters, etc. pages in the new IA. So the corresponding content on that monolithic configuration page will be broken out into separate pages, and each page will have the OTel-maintained components listed (right now, in a table). WDYT?

Ah, right, yeah, that makes sense

[tiffany76]

Thanks for giving this a look, @chalin!

Have you seen https://opentelemetry.io/status/#collector? The added page content seems to be mostly statuses for components. Might it make sense to put these pages (or at least their status content), under https://opentelemetry.io/status/#collector or a new subsection of https://opentelemetry.io/status/?

Although right now it looks like the new content is mostly adding statuses, the ultimate goal is to bridge the gap between component documentation and the official docs. As we iterate, we will hopefully be able to bring in more of the component metadata than just the stability level. So I don't think we should move this content to the /status page.

Re. the automation: under which repo will the GH actions run (and where will be the associated scripts, if any)?

I think @jaydeluca can answer that better than me.

There is https://opentelemetry.io/docs/collector/distributions/. Is there an overlap with what is being proposed here as content? If so, will it lead to confusion for readers?

No, there shouldn't be much overlap. The current Distributions page will be reworked in the new IA to help users focus on choosing a distribution. But I don't think we'll be going into component-level detail there. The only overlap I see (as of now, that could change, of course) is if we indicate which distributions each component is part of, as recommended by Damien.

[svrnm]

Thanks, this is really interesting and going to be helpful. One comment/ask:

While we still have it, can this script populate registry entries first and then fill the pages with that information?

[jaydeluca]

thank you everyone for all the valuable feedback thus far, I'm excited to see what we can do here. Apologies for the delay, I was traveling this week.

Here is my plan for next steps, as of now:

• Work on reworking the page copy, including taking out stability verbiage and link to existing resources for that information
• Include more details about the other distributions and which ones the components are included in (comment)
• Remove the x- components
• Explore integration with the existing registry as well
• Rework things a bit so the automation just manages the list contents on the pages instead of generating the entire page content

As for the tooling itself, I have everything running in https://github.com/jaydeluca/collector-watcher for now, which I think makes sense so I can iterate quickly until I get things in a good place and working - unless there are others who would like to collaborate and be included in code reviews, I'm happy to do whatever makes sense. I can also rewrite it in another language if that makes it easier for others to get involved, although it's not too complex.

My thinking is that I can get everything working end to end in an automated fashion from that repo and let it soak for a bit, monitor that it works as we want, and then we could move it either into this repo or if we want to keep it as a separate repo within the OTel org.

So the goal is that once these pages are finalized and merged, every night the automation will run, detect if there are updates, and will open PR's automatically.

Keep the feedback coming if you have other ideas or concerns. I will hopefully make some progress on this over the next week. Thanks all!