1

Hi i have create a cluster Redis with sentinel composed by 3 aws instances, i have configured sentinel to have an HA redis cluster and work, but if i simulate a crash of master (shutdown of master instance), sentinel installed on slaves, not locate sentinel of master and the election fail. My sentinel configuration is:

sentinel monitor master ip-master 6379 2
sentinel down-after-milliseconds master 5000
sentinel failover-timeout master 10000
sentinel parallel-syncs master 1

Same file to all instaces

stecog
  • 2,202
  • 4
  • 30
  • 50
  • It is clear what your layout is. Do you have: a) 3 nodes, w/sentinel on each and one w/master another w/slave? b) 1 master node, 3 slaves each running sentinel as well? c) Something else? If you are running sentinel on the same node as the master, how are you simulating the failure - by stopping the node, using the debug command on the master, stopping the redis process, or something else? Have you checked the logs of the Sentinel, and what do they say? – The Real Bill Oct 13 '15 at 14:54
  • I have 1 master node and 2 slave (total 3 servers), at each node sentinel is installed with the same configuration file. To simulate failover i tried in two way, a) shutdown the redis service on master (election work) b) phisical shutdown master, `halt`command (election not work) – stecog Oct 13 '15 at 15:33

3 Answers3

1

There are issues when running sentinel on the same node as the master and attempting to trigger a failover. Try it w/o running Sentinel on the master. Ultimately this means not running Sentinel on the same nodes as the Redis instances.

In your case your dead-node simulation is showing why you should not run Sentinel on the same node as Redis: If the node dies you lose one of your sentinels. In theory it should still work but as you and others have seen it isn't certain to work. I have some theories why but I've not yet confirmed them.

In a sense Sentinel is partly a monitoring system. Running a monitoring solution on the same nodes as are being monitored is generally unadvisable anyway, so you should be using off-node sentinels anyway. As Sentinel is resource efficient you don't necessarily need dedicated machines or large VMs. Indeed if you have a static set of application servers (where your client code runs), you should run Sentinel there, keeping in mind you want 3 minimum and a quorum of 50%+1.

The Real Bill
  • 14,884
  • 8
  • 37
  • 39
  • But if install sentinel out of node, how to make a configuration file of sentinel? Autodiscover work because sentinel of slave call master, if sentinel is not installed on slave node, how does the master know who is the slave ? – stecog Oct 14 '15 at 16:02
  • The master knows who is a slave no by talking to Sentinel, but by what clients have connected to it and issues the sync command. You configure Sentinel exactly the same way - by pointing it at the master IP:PORT. Sentinel figures out the rest by asking the master. – The Real Bill Oct 14 '15 at 16:04
  • Ops, sorry...you are right, then it would be enough to install sentinel in other 3 machines that are not the redis instances – stecog Oct 14 '15 at 16:06
  • Precisely so. And ensure the "old" sentinels are gone. ;) Ideally you want them to be on the same network as your client code because it tests the access route. But ultimately they can be anywhere other than on the Redis nodes also long as they can connect to the Redis instances. – The Real Bill Oct 14 '15 at 16:08
  • 1
    To the client is not possible because they are in autoscaling group and are destroyed and recreated Continuously. I installed to other instance of network. Thanks – stecog Oct 14 '15 at 16:15
1

recent redis version introduced the "protected-mode" option, which defaults to yes.

with protected-mode set to yes, redis instances, without a password set will not allow remote clients to execute commands.

this also affects sentinels master election.

try it with setting "protected-mode no" in the sentinels. this will allow them to talk to each other.

pcmanprogrammeur
  • 307
  • 2
  • 9
0

If you don't want to set protected-mode as no. you'd better set masterauth myredis in redis.conf and use sentinel auth-pass mymaster myredis in sentinel.conf

Kiah Han
  • 83
  • 1
  • 1
  • 5