receive: Add retry-after backoff with jitter via header field to active-series-limiting #8356
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
roth-wine
force-pushed
the
pr_add_retry_after_to_active_series_limiting
branch
2 times, most recently
from
643515c to
fe113bf
Compare
roth-wine
force-pushed
the
pr_add_retry_after_to_active_series_limiting
branch
from
fe113bf to
80ce0bc
Compare
|
As far is a can see in the contributions for the thanos receive active series limiting stuff, maybe @GiedriusS or @saswatamcode can have a look at this? |
roth-wine
force-pushed
the
pr_add_retry_after_to_active_series_limiting
branch
from
08c0f07 to
f13031b
Compare
…ve-series-limiting By default, the active-series-limiting does only return a 429 when the tenant exceeded the limit. Prometheus does parse the retry-after header by default¹ and if the header is not set, it will do not set a default backoff on prometheus side. This results in a instant retry of sending the data again which may result in a potential increase of requests. ¹https://github.com/prometheus/prometheus/blob/bfbae39931a6ddeb5913a6ea8d48a34bbebc6d29/storage/remote/client.go#L309 Signed-off-by: roth-wine <[email protected]>
roth-wine
force-pushed
the
pr_add_retry_after_to_active_series_limiting
branch
from
f13031b to
b280f42
Compare
|
@saswatamcode I have adjusted everything we have discussed before and rebased the branch to get this thing merged. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
By default, the active-series-limiting does only return a 429 when the tenant exceeded the limit. Prometheus does parse the retry-after header by default¹ and if the header is not set, it will do not set a default backoff on prometheus side. This results in a instant retry of sending the data again which may result in a potential increase of requests.
¹https://github.com/prometheus/prometheus/blob/bfbae39931a6ddeb5913a6ea8d48a34bbebc6d29/storage/remote/client.go#L309