Mimir distributor to ingester network x20

[garnus]

Hello,
I'm using kube-prometheus-stack with mimir-distributed. Prometheus send about 1MB/s to mimir-distributor how makes x8 this traffic to 3 ingesters. Why?

`mimir:
structuredConfig:
common:
storage:
backend: s3
s3:
bucket_name: tst-prometheus
endpoint: AAA
access_key_id:YYY
secret_access_key: XXX
region: auto
insecure: false
limits:

retention_period: 48h

compactor_blocks_retention_period: 48h
ingester:
replicas: 1
persistentVolume:
enabled: false
alertmanager:
persistentVolume:
enabled: false
compactor:
downsampling_enabled: true
persistentVolume:
enabled: false
minio:
enabled: false
store_gateway:
replicas: 3
persistentVolume:
enabled: false
gateway:
enabledNonEnterprise: true
replicas: 2
nginx:
config:
enableIPv6: false
alertmanager:
enabled: false
ruler:
enabled: false
nginx:
enabled: false
runtimeConfig:
ingester_limits:
max_ingestion_rate: 300000
max_series: 1500000
max_tenants: 1000
max_inflight_push_requests: 30000
distributor_limits:
max_ingestion_rate: 4000000
max_inflight_push_requests: 30000
max_inflight_push_requests_bytes: 50000000
overrides:
promek:
ingestion_rate: 50000
ingestion_tenant_shard_size: 9
max_global_series_per_user: 1500000
max_fetched_series_per_query: 100000`

[deniszh]

It's by design, I'm afraid. GRPC by default uncompressed, incoming protobuf is compressed with snappy and distributors do 3x replication.
Try to configure GRPC compression, gzip for good compression, or S2 for good balance between performance and compression. See #8522 and #10411

[colega]

In addition to @deniszh's answer, please take a look at their awesome article about saving costs by enabling compression. 👏

[garnus]

Thank you :)

distributor:
  extraArgs:
    ingester.client.grpc-compression: gzip
  replicas: 3

Works great.