0

When deploying Redis sentinel in the kubernetes cluster, if the underlying k8s node that hosts the Redis master node restarts, sentinel is unable to stabilize on a new master. It keeps on cycling masters and gives the below error. RedisReadOnlyException: READONLY You can’t write against a read only replica.

Below are the sentinel logs from the container that are repeating:

try-failover master mymaster redis-node-3.redis-headless.test.svc.cluster.local 6379 failover-abort-not-elected master mymaster new-epoch 731 vote-for-leader Next failover delay: I will not start a failover before Thu May 4

The system goes into a state were we’re not able to do any write operations until all the nodes are restarted.

Any pointers on how to handle this would be helpful.

The underlying kubernetes worker node that is hosting the redis master node got restarted that caused this problem.

danronmoon
  • 3,814
  • 5
  • 34
  • 56
John
  • 1
  • share your simple code with that you are trying to connect. Hope you are checking and getting the master & slave IP each time else it could be like you are writing slave. – Harsh Manvar May 10 '23 at 15:56
  • just for ref check my code are you doing same ? https://stackoverflow.com/a/70271427/5525824 – Harsh Manvar May 10 '23 at 17:02
  • yes, we're writing only to the master node. This is the test code we're using to test this from redisclient export password=$(cat redis-password/redis-password) && redis-cli --tls --cert ./certs/tls.crt --key ./certs/tls.key --cacert ./certs/ca.crt -h $(echo $(redis-cli --tls --cert ./certs/tls.crt --key ./certs/tls.key --cacert ./certs/ca.crt -h redis -p 26379 -a $password sentinel masters) | awk '{print $4}') -p 6379 -a $password set foo bar && redis-cli --tls --cert ./certs/tls.crt --key ./certs/tls.key --cacert ./certs/ca.crt -h redis -p 26379 -a $password sentinel masters – John May 10 '23 at 17:11

0 Answers0