agentgateway: report nacks via gateway status #12770
Open
+787
−12
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Joe McGuire <[email protected]> Signed-off-by: Joe McGuire <[email protected]>
Signed-off-by: Joe McGuire <[email protected]> Signed-off-by: Joe McGuire <[email protected]>
Signed-off-by: Joe McGuire <[email protected]> Signed-off-by: Joe McGuire <[email protected]>
Signed-off-by: Joe McGuire <[email protected]> Signed-off-by: Joe McGuire <[email protected]>
Signed-off-by: Joe McGuire <[email protected]> Signed-off-by: Joe McGuire <[email protected]>
…an limit the perf damage we're doing here Signed-off-by: Joe McGuire <[email protected]> Signed-off-by: Joe McGuire <[email protected]>
Signed-off-by: Joe McGuire <[email protected]> Signed-off-by: Joe McGuire <[email protected]>
Signed-off-by: Joe McGuire <[email protected]> Signed-off-by: Joe McGuire <[email protected]>
Signed-off-by: Joe McGuire <[email protected]> Signed-off-by: Joe McGuire <[email protected]>
Signed-off-by: Joe McGuire <[email protected]> Signed-off-by: Joe McGuire <[email protected]>
Signed-off-by: Joe McGuire <[email protected]> Signed-off-by: Joe McGuire <[email protected]>
Signed-off-by: Joe McGuire <[email protected]> Signed-off-by: Joe McGuire <[email protected]>
Signed-off-by: Joe McGuire <[email protected]>
Signed-off-by: Joe McGuire <[email protected]>
Signed-off-by: Joe McGuire <[email protected]>
github-actions
bot
added
kind/feature
Signed-off-by: Joe McGuire <[email protected]>
Signed-off-by: Joe McGuire <[email protected]>
npolshakova
reviewed
Oct 29, 2025
npolshakova
reviewed
Oct 29, 2025
|
|
||
| // onAck publishes an ACK event as a k8s event. | ||
| func (p *Publisher) onAck(event AckEvent) { | ||
| recoveredNackID := ComputeNackID(event.Gateway.Namespace+"/"+event.Gateway.Name, event.TypeUrl) |
There was a problem hiding this comment.
I think this is getting computed twice? Is it different from AnnotationNackID?
There was a problem hiding this comment.
I think we have to compute it again here to know which nack this would resolve since theoretically we could have more than one nack at a time on the same gateway with different TypeURL
npolshakova
reviewed
Oct 29, 2025
npolshakova
reviewed
Oct 29, 2025
| } | ||
|
|
||
| // onNack publishes a NACK event as a k8s event. | ||
| func (p *Publisher) onNack(event NackEvent) { |
There was a problem hiding this comment.
How does this work for multiple gateways (like multiple replicas of one k8s Gateway resource?)
There was a problem hiding this comment.
We would report multiple events (one for each replica which encountered a nack), but they ultimately all have the same involved object, so the gateway would still have the programmed false condition only added once.
There was a problem hiding this comment.
Actually now that we've switched to event recorder we should only create one event per gateway here
npolshakova
reviewed
Oct 29, 2025
Signed-off-by: Joe McGuire <[email protected]>
…emaining nacks Signed-off-by: Joe McGuire <[email protected]>
jmcguire98
force-pushed
the
report-agw-nacks-on-gateways
branch
from
d1de508 to
d93746e
Compare
Signed-off-by: Joe McGuire <[email protected]>
Signed-off-by: Joe McGuire <[email protected]>
Signed-off-by: Joe McGuire <[email protected]>
Signed-off-by: Joe McGuire <[email protected]> Signed-off-by: Joe McGuire <[email protected]>
Signed-off-by: Joe McGuire <[email protected]>
Signed-off-by: Joe McGuire <[email protected]>
…ing it Signed-off-by: Joe McGuire <[email protected]>
Signed-off-by: Joe McGuire <[email protected]>
Signed-off-by: Joe McGuire <[email protected]>
Signed-off-by: jmcguire98 <[email protected]>
Signed-off-by: Joe McGuire <[email protected]>
Signed-off-by: Joe McGuire <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
We want to be able to more easily surface to end users cases where configuration they have provided has been translated into config that was rejected (NACKed) by the gateway they were sending it to. In the past this most frequently happened when a user created a policy with illegal CEL which was nacked at the gateway (subsequently we added CEL validation before translating).
Achieving status reporting here is complicated by the fact that kgateway can be deployed with replicas and leader election enabled. Currently we only allow the leader to write to status, and this works because the leader does translate every resource passed into it. However, we can't follow the exact same pattern for NACK reporting because only the replica with an active xDS connection to a specific gateway can detect NACKs from that gateway (the leader may not have
connections to all gateways).
So, we needed a mechanism to surface NACKs across replicas in order for the leader to adequately report status across all gateways. To achieve this I have added a NACK publisher that publishes k8s events when an instance of a NACK occurs, and publishes a subsequent ACK event when the issue causing that NACK is resolved.
This enables the leader to report NACK statuses via a krt collection of events (filtered to events specifically involving gateways to lighten performance costs).
Change Type
/kind new_feature
Changelog
Additional Notes
fixes #12671