Establish/Document a UDS Core Observability Reference Architecture

[joelmccoy]

Is your feature request related to a problem? Please describe.

UDS Core lacks a single, documented “golden path” for observability across metrics, logs, dashboards, and alert routing.

Today:

• Metrics → Prometheus/Alertmanager alerts can be declared via Prometheus CRDs or written as Grafana alerts
• Logs → Loki alerting has multiple competing approaches (Loki ruler, Grafana alerts via configmaps/overrides/operator, manual UI) (see Feature Request: Loki-based Alerting Support in UDS Core #1845)

It’s unclear where users should declare rules, how those rules should be loaded, where the end user should view those alerts, how alerts should be routed externally, and which Kubernetes resources/CRDs are the source of truth.

This ambiguity makes it harder to optimize UDS Core around a consistent pipeline and increases the learning curve for end users.

Describe the solution you'd like

• Given a UDS Core deployment
• When a user needs to: (a) create dashboards, (b) create metrics alerts, (c) create log alerts, (d) route alerts to external systems
• Then they follow the documented reference architecture for defining/managing these things

Describe alternatives you've considered

• Leave it up to the end user on how they want to configure the observability stack

Additional context

N/A

[joelmccoy]

I think a diagram of the recommend flow for the observability stack would be helpful. Here is what I would propose for the overall opinions:

Metrics alerts

• Declare with PrometheusRule (CRD, monitoring.coreos.com).
• Evaluate in Prometheus.

Logs alerts

• Declare Loki alerting rules as files delivered via ConfigMaps (labeled and reloaded by a Loki ruler sidecar).
• Evaluate in Loki Ruler.
• Route to Alertmanager (unified alert pipeline).

Dashboards

• Declare as ConfigMaps (labeled for Grafana config reloader) as the default GitOps path.

External alert routing

• Prefer Alertmanager receivers (Slack, PagerDuty, Email, Webhook) configured via AlertmanagerConfig in-cluster via helm overrides.

Governance & tenancy

• Namespace-scoped ownership of rules/dashboards.

[joelmccoy]

Some good questions to help provide answers/documentations for from @ntwkninja around alerting:

Will only critical alerts be configured by default? Do I have a way to opt into some things that are available / recommend but may not be critical? Are alert categories defined?

What is the plan for ensuring alerts aren't too noisy? For example, if there are too many notifications to email or chat team members will likely mute channels or create email inbox folders and rules to create clarity.

Can I organize which alerts go where (i.e. if I have email and chat capabilities, I can do both for critical alerts but may only have chat for some of the opt in pieces if that is a provided capability)?

We should be able to answer a lot of these questions with documentation/best practices. As all of these things can be addressed with proper Alertmanager Configuration for you envrionment.

[joelmccoy]

Maybe consult UX folks on where to put/structure of this reference arch.