When deploying Redis sentinel in the kubernetes cluster, if the underlying k8s node that hosts the Redis master node restarts, sentinel is unable to stabilize on a new master. It keeps on cycling masters and gives the below error. RedisReadOnlyException: READONLY You can’t write against a read only replica.
Below are the sentinel logs from the container that are repeating:
try-failover master mymaster redis-node-3.redis-headless.test.svc.cluster.local 6379 failover-abort-not-elected master mymaster new-epoch 731 vote-for-leader Next failover delay: I will not start a failover before Thu May 4
The system goes into a state were we’re not able to do any write operations until all the nodes are restarted.
Any pointers on how to handle this would be helpful.
The underlying kubernetes worker node that is hosting the redis master node got restarted that caused this problem.