Update state when batch dropped

[duhminick]

Description of the issue

The Amazon CloudWatch Agent currently only updates the state file when it successfully pushes a batch of logs to the CloudWatch Logs service. When a "poison pill" batch fails after exhausting all retry attempts, the state file is not updated. This causes the agent to reprocess the same problematic batch after restart, potentially creating an infinite loop of failed attempts.

Description of changes

This PR implements poison pill handling for the CloudWatch Logs output plugin to improve agent resilience:

Key Changes:

• Modified logEventBatch to separate state-updating callbacks from success callbacks
• Added updateState() method that only executes state file updates without other success metrics
• Updated sender.Send() to call updateState() when retry attempts are exhausted

How it works:

1. When a batch fails after exhausting all retry attempts, it's identified as a poison pill
2. The agent updates the state file to mark the batch's range as processed (preventing reprocessing)

This approach leverages the existing retry mechanism without modification and ensures the agent moves past problematic log entries after restart.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

1. Unit tests
2. Integration tests

Requirements

Before commiting your code, please do the following steps.

1. Run make fmt and make fmt-sh
2. Run make lint

---

Integration Tests

To run integration tests against this PR, add the ready for testing label.

[okankoAMZ]

why are we iterating backwards?

[duhminick]

Just following the same pattern for the done() function

amazon-cloudwatch-agent/plugins/outputs/cloudwatchlogs/internal/pusher/batch.go

Lines 179 to 186 in 73da7a4

	func (b *logEventBatch) done() {
	b.updateState()
	for i := len(b.doneCallbacks) - 1; i >= 0; i-- {
	done := b.doneCallbacks[i]
	done()
	}
	}

[the-mann]

where are the integ tests? or did you mean you were relying on the existing integ tests

[the-mann]

nit – why not use range?

[duhminick]

Existing integ tests and replied to the same question above

[github-actions]

This PR was marked stale due to lack of activity.