when the master goes down the other nodes stays slaves and I am not able to write. I am using HAProxy to connect to the redis nodes,
I thought one of the node from the slves will become master and hte web app will be able to write to it, but the slaves stay slaves, and I am not able to write to them
A failover can only occur when a quorum between sentinels can be reached. A quorum in this context is the number of sentinels that must agree that the master is down (or otherwise not reachable) before a failover can occur.
Once a quorum is reached, one of the sentinels of the quorum will be selected to start the failover process. That sentinel asks all the connected sentinels for authorization to perform the failover. A majority of sentinels must authorize the failover for it to happen.
The quorum required to start a failover is configurable. When you register a sentinel to monitor a master, you must give it the quorum value: sentinel monitor <master-group-name> <ip> <port> <quorum>. This line should be the same in each of your sentinels. You can change the quorum at run time by running this on each of the sentinels: SENTINEL SET objects-cache-master quorum new_quorum_num.
To check the current quorum, log into one of your sentinels and run the ckquorum command. You will need your master name that the sentinels monitor. You can retrieve a list of monitored masters with:
sentinel masters
and looking for the "Name" field of the master you need in the results. Once you have the master name, you can check the quorum:
sentinel ckquorum master_name
The results will look like OK 3 usable Sentinels. Quorum and failover authorization can be reached if a quorum can be reached.` If you get a message that a quorum can't be reached, a failover can't be started.
The value you use for the quorum is important, and there are some guidelines from the docs on how to set it:
From the Algorithms and Internals section (https://redis.io/topics/sentinel#quorum):
- If a the quorum is set to a value smaller than the majority of Sentinels we deploy, we are basically making Sentinel more sensitive to master failures, triggering a failover as soon as even just a minority of Sentinels is no longer able to talk with the master.
- If a quorum is set to a value greater than the majority of Sentinels, we are making Sentinel able to failover only when there are a very large number (larger than majority) of well connected Sentinels which agree about the master being down.
