ntp collector's data is not consistent as expected

[gnnx0425]

What did you expect to see?

Node exporter is running as a systemd service on Linux VM. --collect.ntp is enabled to collector NTP related metrics. I expect to see the value of the metrics is consistent with what I obtained from system command.

What did you see instead?

MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* 117-host>     2  10   377   521   -798us[ -633us] +/-   27ms

chronyc sourcestats
Name/IP Address            NP  NR  Span  Frequency  Freq Skew  Offset  Std Dev
==============================================================================
117-host.>                      29  19  297m     +0.009      0.033  **-1201us**   282us

# HELP node_ntp_offset_seconds ClockOffset between NTP and local clock.
# TYPE node_ntp_offset_seconds gauge
node_ntp_offset_seconds **-5.257e-06**

The offset value from "chronyc sourcestats" is not consistent from node exporter node_ntp_offset_seconds metrics value.

[SuperQ]

Correct, that's not what the NTP collector does or is useful for. It's basically a blackbox ping using NTP protocol. It doesn't understand Chrony's protocol.

You want the chrony_exporter for that.

[gnnx0425]

Sorry, I'm a little confuse. For my undestanding, Chronyd and ntpd are the implementation of NTP protocal. I noticed that in ntp collector source code, https://github.com/beevik/ntp/ is imported. Why does NTP collector not know Chrony's protocal?

[SuperQ]

Chrony has its own protocol, which is not NTP, for accessing chronyd statistics.

See the facebook/time package. Which is what is used by the chrony_exporter.

See also the Command and monitoring access chrony documentation.

[robbat2]

Can you please direct users of the ntp collector to https://github.com/sapcc/ntp_exporter instead?

It's close to a usable drop-in, with the note that node_offset_seconds is now ntp_drift_seconds

This is a minor loss because node_exporter provides release binaries for many Linux architectures (esp ARM & MIPS), whereas ntp_exporter does not. Having a single exporter is also beneficial on space-constrained systems.

[robbat2]

The NTP collector does work correctly on ntpsec/ntpd, but I agree with the criticism that it's basically a blackbox view into the local ntpd instance.

[Whissi]

Be aware that ntp_exporter is supposed to compare local system time against a remote reference timeserver you set in ntp_exporter.

This will increase network traffic and NTP server load of set reference timeserver by a lot given that ntp_exporter will query configured reference timeserver every $scrape_interval (node_exporter's time module queried local running NTP server by default and acted on reported data instead).

I wonder if it would be enough if node_exporter would report same like systemd's timedatectl which shows

System clock synchronized: yes/no

based on https://github.com/systemd/systemd/blob/v253/src/timedate/timedated.c#L560-L570

It isn't an option if you like to monitor and graph offset (in which case you probably should run dedicated exporter for your time service) but it should be enough for basic monitoring to get alerted when nothing adjusted time and/or buf.maxerror is too high which indicates a large offset.

[robbat2]

@Whissi the approx 1:1 conversion is running ntp_exporter on the same host as node_exporter, which doesn't have the traffic issue any more than the present code (that scrapes against 127.0.0.1 or other collector.ntp.server configured target.

As for the timedatectl, it doesn't need NTP at all, and the metric is already available:
https://github.com/systemd/systemd/blob/main/src/timedate/timedated.c#L560-L570
adjtimex maxerror < 16 seconds.

Which would map to an alert of:
node_timex_maxerror_seconds > 16

Yes, systemd doesn't check for anything beyond that.

node_timex_status

For my own monitoring of local time, I have all of these:

          // node_timex_sync_status specifies if it thinks it's got a sync source.
          // It does NOT require ntp data!
          'node_timex_sync_status{%(instance_filter)s} != 1',
          // This variant came from upstream docs:
          'min_over_time(node_timex_sync_status{%(instance_filter)s}[5m]) == 0 and node_timex_maxerror_seconds{%(instance_filter)s} >= 16',

          // see man adjtimex(2) timex.status
          // read /usr/include/linux/timex.h
          // or /usr/include/bits/timex.h
          // Good combined states:
          // 0x2001 STA_NANO|STA_PLL
          // 0x6001 STA_NANO|STA_PLL|STA_MODE
          // Bad bitflags:
          // TODO: how to get bitwise flags in Prom?
          // 0x0040 STA_UNSYNC
          // 0x0200 STA_PPSJITTER
          // 0x0400 STA_PPSERROR
          // 0x0800 STA_CLOCKERR
node_timex_status{%(instance_filter)s} and node_timex_status != 0x2001 and node_timex_status != 0x6001


          'min_over_time(node_ntp_sanity{%(instance_filter)s}[1h]) != 1',
          'node_ntp_stratum{%(instance_filter)s} and (node_ntp_stratum{%(instance_filter)s} < 2 or node_ntp_stratum{%(instance_filter)s} > 5)',
          'node_ntp_reference_timestamp_seconds{%(instance_filter)s} < 1e9',
          'node_ntp_offset_seconds{%(instance_filter)s} and (abs(node_ntp_offset_seconds{%(instance_filter)s}) > %(threshold_node_ntp_offset_seconds)s)',
          'node_ntp_drift_seconds{%(instance_filter)s} and (abs(node_ntp_drift_seconds{%(instance_filter)s}) > %(threshold_node_ntp_offset_seconds)s)',
          'node_ntp_root_delay_seconds{%(instance_filter)s} > %(threshold_node_ntp_root_delay_seconds)s',
          'node_ntp_root_dispersion_seconds{%(instance_filter)s} > %(threshold_node_ntp_root_dispersion_seconds)s',