Redis Sentinel master not downgraded to slave immediately

Question

I have an architecture with three Redis instances (one master and two slaves) and three Sentinel instances. In front of it there is a HaProxy. All works well until the master Redis instance goes down. The new master is properly chosen by Sentinel. However, the old master (which is now down) is not reconfigured to be a slave. As a result, when that instance is up again I have two masters for a short period of time (about 11 seconds). After that time that instance which was brought up is properly downgraded to slave.

Shouldn't it work that way, that when the master goes down it is downgraded to slave straight away? Having that, when it was up again, it would be slave immediately. I know that (since Redis 2.8?) there is that CONFIG REWRITE functionality so the config cannot be modified when the Redis instance is down.

Having two masters for some time is a problem for me because the HaProxy for that short period of time instead of sending requests to one master Redis, it does the load balancing between those two masters.

Is there any way to downgrade the failed master to slave immediately?

Obviously, I changed the Sentinel timeouts.

Here are some logs from Sentinel and Redis instances after the master goes down:

Sentinel

81358:X 23 Jan 22:12:03.088 # +sdown master redis-ha 127.0.0.1                       63797.0.0.1 26381 @ redis-ha 127.0.0.1 6379
81358:X 23 Jan 22:12:03.149 # +new-epoch 1
81358:X 23 Jan 22:12:03.149 # +vote-for-leader 6b5b5882443a1d738ab6849ecf4bc6b9b32ec142 1
81358:X 23 Jan 22:12:03.174 # +odown master redis-ha 127.0.0.1 6379 #quorum 3/2
81358:X 23 Jan 22:12:03.174 # Next failover delay: I will not start a failover before Sat Jan 23 22:12:09 2016
81358:X 23 Jan 22:12:04.265 # +config-update-from sentinel 127.0.0.1:26381 127.0.0.1 26381 @ redis-ha 127.0.0.1 6379
81358:X 23 Jan 22:12:04.265 # +switch-master redis-ha 127.0.0.1 6379 127.0.0.1 6381
81358:X 23 Jan 22:12:04.266 * +slave slave 127.0.0.1:6380 127.0.0.1 6380 @ redis-ha 127.0.0.1 6381
81358:X 23 Jan 22:12:04.266 * +slave slave 127.0.0.1:6379 127.0.0.1 6379 @ redis-ha 127.0.0.1 6381
81358:X 23 Jan 22:12:06.297 # +sdown slave 127.0.0.1:6379 127.0.0.1 6379 @ redis-ha 127.0.0.1 6381

Redis

81354:S 23 Jan 22:12:03.341 * MASTER <-> SLAVE sync started
81354:S 23 Jan 22:12:03.341 # Error condition on socket for SYNC: Connection refused
81354:S 23 Jan 22:12:04.265 * Discarding previously cached master state.
81354:S 23 Jan 22:12:04.265 * SLAVE OF 127.0.0.1:6381 enabled (user request from 'id=7 addr=127.0.0.1:57784 fd=10 name=sentinel-6b5b5882-cmd age=425 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=14 qbuf-free=32754 obl=36 oll=0 omem=0 events=rw cmd=exec')
81354:S 23 Jan 22:12:04.265 # CONFIG REWRITE executed with success.
81354:S 23 Jan 22:12:04.371 * Connecting to MASTER 127.0.0.1:6381
81354:S 23 Jan 22:12:04.371 * MASTER <-> SLAVE sync started
81354:S 23 Jan 22:12:04.371 * Non blocking connect for SYNC fired the event.
81354:S 23 Jan 22:12:04.371 * Master replied to PING, replication can continue...
81354:S 23 Jan 22:12:04.371 * Partial resynchronization not possible (no cached master)
81354:S 23 Jan 22:12:04.372 * Full resync from master: 07b3c8f64bbb9076d7e97799a53b8b290ecf470b:1
81354:S 23 Jan 22:12:04.467 * MASTER <-> SLAVE sync: receiving 860 bytes from master
81354:S 23 Jan 22:12:04.467 * MASTER <-> SLAVE sync: Flushing old data
81354:S 23 Jan 22:12:04.467 * MASTER <-> SLAVE sync: Loading DB in memory
81354:S 23 Jan 22:12:04.467 * MASTER <-> SLAVE sync: Finished with success

This isn't a programming question but a system/server admin one. Serverfault is where to ask this. — The Real Bill, Jan 25 '16 at 03:42
@Damian did you find a solution to your problem. I am also facing the same issue. — akipro, May 23 '16 at 10:42

score 3 · Answer 1 · answered Aug 02 '19 at 18:28

I was also getting the same error when I want to switch master in redis-cluster using sentinel.

+vote-for-leader xxxxxxxxxxxxxxxxxxxxxxxx8989 10495
Next failover delay: I will not start a failover before Fri Aug 2 23:23:44 2019

After resetting sentinel. Cluster works as expected

SENTINEL RESET *

or

SENTINEL RESET mymaster

Run above command in all sentinel server.

davissp14 · Answer 2 · 2016-12-30T02:29:15.137

In the event a Redis node goes down, when/if it recovers, it will recover with the same role it had prior to going down. The Sentinel cannot reconfigure the node if it is unable to ping it. So, there's a brief period of time between the node coming back up and the Sentinel acknowledging and reconfiguring it. This explains the multi-master state.

If you are set on using Haproxy, one workaround would be to reconfigure the Redis node's role prior to starting the process. Redis will boot as a slave as long as there's a SLAVEOF entry in the redis.conf. The primary issue with this workaround is that it doesn't solve network partition scenarios.

Hope that helps.

score 0 · Answer 3 · answered Jul 23 '21 at 21:45

If using HAProxy you can try to query the uptime_in_seconds something like this:

backend redis
    mode tcp
    balance first
    timeout queue 5s
    default-server check inter 1s fall 2 rise 2 maxconn 100
    option tcp-check
    tcp-check connect
    tcp-check send AUTH\ <secret>\r\n
    tcp-check expect string +OK
    tcp-check send PING\r\n
    tcp-check expect string +PONG
    tcp-check send info\ replication\r\n
    tcp-check expect string role:master
    tcp-check send info\ server\r\n
    tcp-check expect rstring uptime_in_seconds:\d{2,}
    tcp-check send QUIT\r\n
    tcp-check expect string +OK
    server redis-1 10.0.0.10:9736
    server redis-2 10.0.0.20:9736
    server redis-3 10.0.0.30:9736

Notice the:

  tcp-check expect rstring uptime_in_seconds:\d{2,}

if uptime is not > 10 seconds, the node will not be added

score 0 · Answer 4 · answered Dec 03 '22 at 13:47

Solution

This can be resolved by making use of the rise option in your HAProxy config.

default-server check inter 1s fall 2 rise 30

# OR

server redis-1 127.0.0.1:6379 check inter 1s fall 2 rise 30

This sets the number of successful checks that must pass for a server to be considered UP. As such this can successfully delay a re-joining Redis node from being considered UP and give Sentinel a chance to change the node's role.

Important Trade-off

The trade-off with this approach, is that your fail-overs will take longer to be respected by HAProxy as you are adding in an extra delay. This delay applies to both your re-joining node after a failure and also your existing slave nodes that are promoted to role:master. Ultimately you will need to make the decision between which option is better for you; having 2 masters momentarily, or taking longer to fail between nodes.

Moerkey · Answer 5 · 2022-12-14T11:20:40.363

If using haproxy a more stable solution would be to check for available slaves. After a reboot, restart or forced switch an old master will still have the role master but no slaves are connected. So the value is zero.

# faulty old master
role:master
connected_slaves:0
slave0:ip=127.0.0.2,port=6379,state=online,offset=507346829,lag=0
slave1:ip=127.0.0.1,port=6379,state=online,offset=507346966,lag=0
master_failover_state:no-failover
...

I would replace

    tcp-check expect string role:master
    tcp-check send info\ server\r\n
    tcp-check expect rstring uptime_in_seconds:\d{2,}

with

tcp-check expect rstring connected_slaves:[^0]

Total config for me.

listen listen-REDIS
        bind 1.1.1.1:6379
        mode tcp
        no option prefer-last-server
        option tcplog
        balance leastconn
        option tcp-check
        tcp-check send "auth STRING\r\n"
        tcp-check send PING\r\n
        tcp-check expect string +PONG
        tcp-check send info\ replication\r\n
        tcp-check expect rstring connected_slaves:[^0]
        tcp-check send QUIT\r\n
        tcp-check expect string +OK
        default-server inter 500ms fall 1 rise 1
        server REDIS01 127.0.0.1:6379 check
        server REDIS02 127.0.0.2:6379 check
        server REDIS03 127.0.0.3:6379 check

Redis Sentinel master not downgraded to slave immediately

5 Answers5

Solution

Important Trade-off