failed pushing to ingester: rpc error: code = Unavailable desc

[kssrao87]

Configured mimir on kubernetes with backend to s3 by changing values.yaml file under https://github.com/grafana/mimir/blob/main/operations/helm/charts/mimir-distributed/values.yaml, trying to push data from prometheus to mimir.
Configuration in prometheus:

remote_write:
      - url: "http://mimir-distributor:8080/api/v1/push"
        send_exemplars: true
        queue_config:
          max_shards: 1
          max_samples_per_send: 500
          capacity: 1000
          batch_send_deadline: 60s
        metadata_config:
          send: false

I am getting below errors in distributor pod:

ts=2023-04-06T10:59:20.115122499Z caller=logging.go:86 level=warn traceID=3a59479d32704c41 msg="POST /api/v1/push (500) 1.025143632s Response: \"failed pushing to ingester: rpc error: code = Unavailable desc = \\n\" ws: false; Content-Encoding: snappy; Content-Length: 191268; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.41.0; X-B3-Sampled: 0; X-B3-Spanid: e9280764f0383de2; X-B3-Traceid: ec00ea61f73f1945e9280764f0383de2; X-Envoy-Attempt-Count: 1; X-Forwarded-Client-Cert: By=spiffe://cluster.local/ns/default/sa/mimir;Hash=764146542e7122999e3e9c24821bceebee232d1d85f277d33a2c636d26977e95;Subject=\"\";URI=spiffe://cluster.local/ns/default/sa/prometheus; X-Forwarded-Proto: http; X-Prometheus-Remote-Write-Version: 0.1.0; X-Request-Id: ee523541-8559-42ef-8f99-e0f98d8aa73f; "
ts=2023-04-06T10:59:26.258320074Z caller=logging.go:86 level=warn traceID=359fb19ef292d359 msg="POST /api/v1/push (500) 1.128399928s Response: \"failed pushing to ingester: rpc error: code = Unavailable desc = \\n\" ws: false; Content-Encoding: snappy; Content-Length: 191268; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.41.0; X-B3-Sampled: 0; X-B3-Spanid: ac60ee0a44215292; X-B3-Traceid: 7a1bf02b69a4df05ac60ee0a44215292; X-Envoy-Attempt-Count: 1; X-Forwarded-Client-Cert: By=spiffe://cluster.local/ns/default/sa/mimir;Hash=764146542e7122999e3e9c24821bceebee232d1d85f277d33a2c636d26977e95;Subject=\"\";URI=spiffe://cluster.local/ns/default/sa/prometheus; X-Forwarded-Proto: http; X-Prometheus-Remote-Write-Version: 0.1.0; X-Request-Id: a82c9c60-8001-4bea-afb3-23d04314f1a0; "

I am using istio, I thought because of that I got the above error messages, after disabling istio able to push the data to distributor from prometheus, but I am unable to see the data in s3 store.I didn't see any error logs in distributor, ingester, store-gateway or compactor.

Logs of mimir-ingester-0:

ts=2023-04-07T05:47:27.626116186Z caller=spanlogger.go:85 user=anonymous level=debug event="start commit" succeededSamplesCount=500 failedSamplesCount=0 succeededExemplarsCount=0 failedExemplarsCount=0
ts=2023-04-07T05:47:27.626418335Z caller=spanlogger.go:85 user=anonymous level=debug event="complete commit" commitDuration=211.359µs
ts=2023-04-07T05:47:27.663908607Z caller=spanlogger.go:85 user=anonymous level=debug event="acquired append lock"
ts=2023-04-07T05:47:27.664042554Z caller=spanlogger.go:85 user=anonymous level=debug event="got appender for timeseries" series=500
ts=2023-04-07T05:47:27.667879076Z caller=spanlogger.go:85 user=anonymous level=debug event="start commit" succeededSamplesCount=500 failedSamplesCount=0 succeededExemplarsCount=0 failedExemplarsCount=0
ts=2023-04-07T05:47:27.668343741Z caller=spanlogger.go:85 user=anonymous level=debug event="complete commit" commitDuration=334.974µs
ts=2023-04-07T05:47:27.695434609Z caller=spanlogger.go:85 user=anonymous level=debug event="acquired append lock"
ts=2023-04-07T05:47:27.695547724Z caller=spanlogger.go:85 user=anonymous level=debug event="got appender for timeseries" series=500
ts=2023-04-07T05:47:27.700379155Z caller=spanlogger.go:85 user=anonymous level=debug event="start commit" succeededSamplesCount=500 failedSamplesCount=0 succeededExemplarsCount=0 failedExemplarsCount=0
ts=2023-04-07T05:47:27.701410893Z caller=spanlogger.go:85 user=anonymous level=debug event="complete commit" commitDuration=927.654µs
ts=2023-04-07T05:47:27.745828293Z caller=spanlogger.go:85 user=anonymous level=debug event="acquired append lock"

when I search for stor related logs in ingester logs:

k logs mimir-ingester-0 | grep stor
ts=2023-04-07T05:29:04.988062306Z caller=sanity_check.go:39 level=info msg="Checking object storage config"
ts=2023-04-07T05:29:05.719354584Z caller=sanity_check.go:44 level=info msg="Object storage config successfully checked"
ts=2023-04-07T05:30:05.759668306Z caller=ingester.go:2024 level=debug msg="shipper successfully synchronized TSDB blocks with storage" user=anonymous uploaded=0
ts=2023-04-07T05:30:07.618693842Z caller=broadcast.go:48 level=debug msg="Invalidating forwarded broadcast" key=collectors/store-gateway version=6 oldVersion=5 content=[mimir-store-gateway-0] oldContent=[mimir-store-gateway-0]
ts=2023-04-07T05:31:05.664145985Z caller=ingester.go:2024 level=debug msg="shipper successfully synchronized TSDB blocks with storage" user=anonymous uploaded=0
ts=2023-04-07T05:32:05.629455114Z caller=ingester.go:2024 level=debug msg="shipper successfully synchronized TSDB blocks with storage" user=anonymous uploaded=0
ts=2023-04-07T05:33:05.590536951Z caller=ingester.go:2024 level=debug msg="shipper successfully synchronized TSDB blocks with storage" user=anonymous uploaded=0
ts=2023-04-07T05:34:05.555499157Z caller=ingester.go:2024 level=debug msg="shipper successfully synchronized TSDB blocks with storage" user=anonymous uploaded=0
ts=2023-04-07T05:35:05.523280214Z caller=ingester.go:2024 level=debug msg="shipper successfully synchronized TSDB blocks with storage" user=anonymous uploaded=0
ts=2023-04-07T05:35:28.901169203Z caller=log.go:194 level=debug msg="Initiating push/pull sync with: mimir-store-gateway-0-1652b28f 172.30.71.209:7946"
ts=2023-04-07T05:36:05.489020776Z caller=ingester.go:2024 level=debug msg="shipper successfully synchronized TSDB blocks with storage" user=anonymous uploaded=0

Any suggestions on why data is not uploaded to s3?

[kssrao87]

Able to see the data in cos

ts=2023-04-07T08:22:59.975721307Z caller=objstore.go:288 level=debug user=anonymous msg="uploaded file" from=/data/tsdb/anonymous/01GXDCYTBRTYYQDBR95J18Y6HE/chunks/000001 dst=01GXDCYTBRTYYQDBR95J18Y6HE/chunks/000001 bucket="tracing: mimir-tsdb"
ts=2023-04-07T08:23:00.280841805Z caller=objstore.go:288 level=debug user=anonymous msg="uploaded file" from=/data/tsdb/anonymous/01GXDCYTBRTYYQDBR95J18Y6HE/index dst=01GXDCYTBRTYYQDBR95J18Y6HE/index bucket="tracing: mimir-tsdb"
ts=2023-04-07T08:23:00.304026942Z caller=ingester.go:2024 level=debug msg="shipper successfully synchronized TSDB blocks with storage" user=anonymous uploaded=1