Skip to content

Conversation

skartikey
Copy link
Contributor

Summary

The chrony plugin was experiencing intermittent panics with error runtime error: index out of range [256] with length 256 when multiple Gather() calls executed concurrently.

Root Cause:
The plugin creates a single fbchrony.Client instance shared across all Gather() calls. When Telegraf triggers a new Gather() before the previous one completes (e.g., when collecting sourcestats with multiple sources), both goroutines
access the shared client concurrently:

  1. Both goroutines call client.Communicate() simultaneously
  2. Both modify n.Sequence++ without synchronization (NOT THREAD-SAFE - Race condition here!)
  3. Both call binary.Write() concurrently (Concurrent writes corrupt memory)
  4. Memory corruption occurs during packet encoding
  5. Panic occurs with index out of range [256]

This race condition is intermittent because it only manifests when gather operations overlap, typically when sourcestats collection takes longer than the configured interval.

Checklist

  • No AI generated code was used in this PR

Related issues

resolves #17757

@telegraf-tiger telegraf-tiger bot added fix pr to fix corresponding bug plugin/input 1. Request for new input plugins 2. Issues/PRs that are related to input plugins labels Oct 6, 2025
The chrony plugin was experiencing intermittent panics with error runtime error: index out of range [256] with length 256 when multiple Gather() calls executed concurrently.
@skartikey skartikey force-pushed the time_chrony_race_fix branch from 759450f to dfe1e78 Compare October 6, 2025 17:06
@carsonsaldanha
Copy link
Contributor

Hi, @skartikey. I have nearly an identical change as this but for the CSV parser in #17573 that's still not been addressed in over a month. A review would be much appreciated. Thanks.

Copy link
Member

@srebhan srebhan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @skartikey! Just one small suggestion...

@srebhan srebhan self-assigned this Oct 8, 2025
@skartikey skartikey requested a review from srebhan October 9, 2025 15:56
@telegraf-tiger
Copy link
Contributor

telegraf-tiger bot commented Oct 9, 2025

Copy link
Member

@srebhan srebhan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @skartikey!

@srebhan srebhan added the ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review. label Oct 9, 2025
@srebhan srebhan assigned mstrandboge and unassigned srebhan Oct 9, 2025
@mstrandboge mstrandboge merged commit d08a4ee into influxdata:master Oct 9, 2025
26 of 27 checks passed
@github-actions github-actions bot added this to the v1.36.3 milestone Oct 9, 2025
srebhan pushed a commit that referenced this pull request Oct 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix pr to fix corresponding bug plugin/input 1. Request for new input plugins 2. Issues/PRs that are related to input plugins ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Panic in chrony input plugin

4 participants