-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Description
We host our Redis pods in K8s
When Redis replicates with a master, Redis uses the pod IP to "name" the replica
Pod IPs are ephemeral, when the pod is bounced, it comes up with a new IP
This is problematic for Redis Sentinel because as pods are bounced, it sees a new IP address for the previous replica and creates a bloated list of "dead replicas"
Redis Sentinel should not have any dead replicas, so the command sentinel reset SHARD_NAME removes them - however executing this after each pod cycle is toilsome
We mitigate this in K8s by provisioning a K8s Service with a ClusterIP
Our helm-chart has an init container which discovers the ClusterIP for the StatefulSet Pod, and sets replica-announce-ip to this value
Redis then advertises itself as being available at that IP address