Ingester - write to WAL: no space left on device

[luvpreetsingh]

Hi,

I keep getting this error,

level=warn ts=2023-04-20T19:52:06.420650274Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=3.952285ms err="user=prod-kube: write to WAL: log samples: write /data/tsdb/prod-kube/wal/00021776: no space left on device" msg=gRPC

That obviously means my ingesters are out of disk space. When I try to increase the disk space it doesn't allow me,

Error: UPGRADE FAILED: cannot patch "mimir-ingester" with kind StatefulSet: StatefulSet.apps "mimir-ingester" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

If the size increase is forbidden, how can I solve this issue?

[luvpreetsingh]

I am using helm chart version 3.0.0

[aknuds1]

This is a Kubernetes specific issue and not really Mimir related, i.e. that you can't directly patch a StatefulSet apart from the fields 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds'. I don't know the procedure for increasing the size of persistent volumes off the top of my head, maybe someone else will chime in?

[luvpreetsingh]

Someone on grafana slack did tell me that this is annoying.

Will this work? The way he said in the 2nd message.

[luvpreetsingh]

I solved by directly editing the pvc using kubectl. Thanks.