Panic in chrony input plugin

[IngmarStein]

Relevant telegraf.conf

[[inputs.chrony]]
  server = "unixgram:///var/run/chrony/chronyd.sock"
  dns_lookup = true
  metrics = ["activity", "tracking", "serverstats", "sources", "sourcestats"]
  socket_group = "wheel"
  socket_perms = "0660"

Logs from Telegraf

2025-08-10T22:07:10Z E! FATAL: [inputs.^@hrony] panicked: runtime error: index out of range [256] with length 256, Stack:
goroutine 483307 [running]:
github.com/influxdata/telegraf/agent.panicRecover(0x140014dd200)
    github.com/influxdata/telegraf/agent/agent.go:1202 +0x60
panic({0x10b46d940?, 0x14000f0d8f0?})
    runtime/panic.go:792 +0x124
encoding/binary.(*encoder).uint8(...)
    encoding/binary/binary.go:792
encoding/binary.(*encoder).value(0x1400161fcf0, {0x10a352000?, 0x140017e6800?, 0x109ef4769?})
    encoding/binary/binary.go:953 +0x920
encoding/binary.(*encoder).value(0x1400161fcf0, {0x10a4c23c0?, 0x140017e671c?, 0x140000d6508?})
    encoding/binary/binary.go:919 +0x880
encoding/binary.(*encoder).value(0x1400161fcf0, {0x10b284c20?, 0x140017e6700?, 0x1010f6100?})
    encoding/binary/binary.go:928 +0x788
encoding/binary.Write({0x138947fc8, 0x1400151c3f8}, {0x10bc85480, 0x1117d6800}, {0x10a972500, 0x140017e6700})
    encoding/binary/binary.go:431 +0x260
github.com/facebook/time/ntp/chrony.(*Client).Communicate(0x14001245758, {0x10bbe36c0, 0x140017e6700})
    github.com/facebook/time@v0.0.0-20240626113945-18207c5d8ddc/ntp/chrony/client.go:37 +0xa0
github.com/influxdata/telegraf/plugins/inputs/chrony.(*Chrony).gatherSourceStats(0x1400149d680, {0x10bc9e260, 0x1400065dbe0})
    github.com/influxdata/telegraf/plugins/inputs/chrony/chrony.go:474 +0x1a8
github.com/influxdata/telegraf/plugins/inputs/chrony.(*Chrony).Gather(0x1400149d680, {0x10bc9e260, 0x1400065dbe0})
    github.com/influxdata/telegraf/plugins/inputs/chrony/chrony.go:160 +0x1d0
github.com/influxdata/telegraf/models.(*RunningInput).Gather(0x140014dd200, {0x10bc9e260, 0x1400065dbe0})
    github.com/influxdata/telegraf/models/running_input.go:260 +0x23c
github.com/influxdata/telegraf/agent.(*Agent).gatherOnce.func1()
    github.com/influxdata/telegraf/agent/agent.go:590 +0x58
created by github.com/influxdata/telegraf/agent.(*Agent).gatherOnce in goroutine 70
    github.com/influxdata/telegraf/agent/agent.go:588 +0xc0

System info

Telegraf 1.35.3, chrony 4.7, macOS 15.6

Docker

No response

Steps to reproduce

1. add the inputs.chrony snippet above to telegraf.conf
2. let it run for a few days
3. eventually observe the panic

Expected behavior

Telegraf should not panic

Actual behavior

The process panics with some data from chrony.

Additional info

No response

[skartikey]

@IngmarStein Thanks for reporting this issue! I've identified that the panic is actually occurring in the underlying facebook/time library that Telegraf uses for chrony communication, not in Telegraf itself.

I've opened a PR to fix this upstream: facebook/time#474

The fix addresses the root cause where binary.Write operations were failing during packet encoding, particularly when collecting sourcestats with DNS lookup enabled (which matches your configuration).

Once that PR is merged and Telegraf updates to the newer version of the facebook/time dependency, this panic should be resolved.

[IngmarStein]

@skartikey I saw the upstream fix - great job! Your explanation also provides a good workaround for me until the fix makes it into a Telegraf release (disabling sourcestats or DNS lookups).

[IngmarStein]

@skartikey looks like it's not just source stats. I recently got another panic when gathering server stats:

2025-08-17T02:14:30Z E! FATAL: [inputs.^@hrony] panicked: runtime error: index out of range [256] with length 256, Stack:
goroutine 353250 [running]:
github.com/influxdata/telegraf/agent.panicRecover(0x14001423440)
    github.com/influxdata/telegraf/agent/agent.go:1202 +0x60
panic({0x10f355940?, 0x14002414948?})
    runtime/panic.go:792 +0x124
encoding/binary.(*encoder).uint8(...) 
    encoding/binary/binary.go:792
encoding/binary.(*encoder).value(0x14001825d40, {0x10e23a000?, 0x140024ab480?, 0x10dddc769?})
    encoding/binary/binary.go:953 +0x920
encoding/binary.(*encoder).value(0x14001825d40, {0x10e3aa360?, 0x140024ab394?, 0x1151b8210?})
    encoding/binary/binary.go:919 +0x880
encoding/binary.(*encoder).value(0x14001825d40, {0x10ed88260?, 0x140024ab380?, 0x2e29a52?})
    encoding/binary/binary.go:928 +0x788
encoding/binary.Write({0x13cae0068, 0x14001163ac8}, {0x10fb6d480, 0x1156be800}, {0x10e85a400, 0x140024ab380})
    encoding/binary/binary.go:431 +0x260
github.com/facebook/time/ntp/chrony.(*Client).Communicate(0x140013d4c78, {0x10facb738, 0x140024ab380})
    github.com/facebook/time@v0.0.0-20240626113945-18207c5d8ddc/ntp/chrony/client.go:37 +0xa0
github.com/influxdata/telegraf/plugins/inputs/chrony.(*Chrony).gatherServerStats(0x140013bbcc0, {0x10fb86260, 0x14000656d00})
    github.com/influxdata/telegraf/plugins/inputs/chrony/chrony.go:298 +0x64
github.com/influxdata/telegraf/plugins/inputs/chrony.(*Chrony).Gather(0x140013bbcc0, {0x10fb86260, 0x14000656d00})
    github.com/influxdata/telegraf/plugins/inputs/chrony/chrony.go:156 +0x17c
github.com/influxdata/telegraf/models.(*RunningInput).Gather(0x14001423440, {0x10fb86260, 0x14000656d00})
    github.com/influxdata/telegraf/models/running_input.go:260 +0x23c
github.com/influxdata/telegraf/agent.(*Agent).gatherOnce.func1()
    github.com/influxdata/telegraf/agent/agent.go:590 +0x58
created by github.com/influxdata/telegraf/agent.(*Agent).gatherOnce in goroutine 90
    github.com/influxdata/telegraf/agent/agent.go:588 +0xc0