too many outstanding requests

[ofiry-tm]

Describe the bug

In grafana, im trying to get data from Mimir for the last 2 days- and its working great.
Then I try to get data from the last 7 days, and I get 429 error.

the logs of Mimir query-frontend pod shows this error while I send the query of the 7-days:

level=error user=anonymous msg="error processing request" try=4 err="too many outstanding requests"
level=error user=anonymous msg="error processing request" try=0 err="context canceled"

the logs of Mimir query pod:

user=anonymous level=warn msg="failed to fetch series" remote=**** err="rpc error: code = Canceled desc = context canceled

What configuration can I change in order to fix this?

Thanks

[pstibrany]

Hi,

if you use query-scheduler component, you can use its -query-scheduler.max-outstanding-requests-per-tenant option to set this value (defaults to 100). If you don't use query-scheduler, then queue in the query-frontend is used instead, and the flag is -querier.max-outstanding-requests-per-tenant (it says "querier" in the name but it's query-frontend that needs to have this flag).

[ofiry-tm]

Hi,

if you use query-scheduler component, you can use its -query-scheduler.max-outstanding-requests-per-tenant option to set this value (defaults to 100). If you don't use query-scheduler, then queue in the query-frontend is used instead, and the flag is -querier.max-outstanding-requests-per-tenant (it says "querier" in the name but it's query-frontend that needs to have this flag).

Thanks, It worked !
at first I tried:

mimir-distributed:
  mimir:
    structuredConfig:      
      query:
        max_outstanding_requests_per_tenant: 10000

But it didnt work, it seems that its not the right place of this config,
but then I enabled the query_scheduler , and added this conf as you suggested:

mimir-distributed:
  mimir:
    structuredConfig:      
      query_scheduler:
        max_outstanding_requests_per_tenant: 10000

and it worked perfectly and now I can make queries of up to 1 year with no issues

[dannyheard7]

FYI for using this option with the query frontend the config is as follows:

mimir-distributed:
  mimir:
    structuredConfig:      
      frontend:
        max_outstanding_per_tenant: 10000

See https://grafana.com/docs/mimir/latest/configure/configuration-parameters/#frontend