Consider handling labels for nagios watchers (e.g. via annotations)

[RamblingCookieMonster]

Apologies for any improper phrasing / terminology, and do please feel free to close this if it does not make sense (in general, or for this specific project), etc.

It might be nice to allow the inclusion of user-specified labels on nagios watchers.

Current state: name and status are included in the prometheus labels, no handling for other labels. ref

Proposed state: name and status are included in the prometheus labels, properties.annotations is parsed to add labels to these prometheus lines, similar to how metric watchers handle this already via properties.labels.

A superficial example:

I configure a choria server with a nagios check:

choria::scout_checks:
  EXAMPLE:
    ...
    properties:
      annotations:
        some_label: EXAMPLE
        another_label: EXAMPLE

The resulting prometheus line might look like this:

choria_machine_nagios_watcher_status{name="EXAMPLE",status="OK",some_label="EXAMPLE",another_label="EXAMPLE"} 0

(I've swapped labels out for annotations, as there is plenty of existing code, and documentation on annotations, they just are not included in the prometheus output yet.)

---

No worries at all if this does not make sense, or ever get done, we can always append labels in telegraf. Cheers!

[ripienaar]

I think the current annotations does this?

https://github.com/choria-io/puppet-choria/blob/0b639b84d12b2e2eccf1b1cfb03ed9a65c68bcab/manifests/scout_check.pp#L25

[RamblingCookieMonster]

Yeah! That's where I started, idea seemed similar to labels, but I think it ends up not being used (or at least, not being used when writing the prometheus format data here)

[ripienaar]

Then perhaps rather than adding another almost identical behaving concept we should just expose annotations to prom?

[RamblingCookieMonster]

💯 makes sense to me, and there are already various docs and code to handle kv pair annotations, just not hitting/integrated into the prometheus data that's written.

(Edited a few spots to reflect this in the issue body)

[ripienaar]

OK, do you feel like doing a PR?

[RamblingCookieMonster]

tldr; probably not

longer: that sounds terse - I can try! Might get stuck at the "get a simple dev environment going to code, test, iterate," given that I'm more a Windows sysadmin without much golang experience, but, will give it a shot.

[ripienaar]

No worries, if you'd rather not that's ok of course but if you'd like to try go for it :)

[RamblingCookieMonster]

Arg, yeah, I can't imagine getting time to work on this for at least a few months. No expectations at all, have a vague notion of how much work goes into OSS projects like this (and this not being the only one you deal with!), do feel free to close this out if it makes sense.

Cheers!

[vjanelle]

@RamblingCookieMonster you can get really far with tests, event for this kind of thing