0

I have a simple redis deployment MASTER, SLAVE and 2 SENTINEL running on docker swarm. I run the stack and all services come up. redis-master start as MASTER and I kill it to test SENTINEL and SLAVE recovering. redis-master then recovers and becomes a new SLAVE. If I ecex into it and run SLAVEOF NO ONE the following happens:

1:M 31 Oct 2019 06:28:32.741 * MASTER MODE enabled (user request from 'id=3907 addr=127.0.0.1:39302 fd=36 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=34 qbuf-free=32734 obl=0 oll=0 omem=0 events=r cmd=slaveof')
1:S 31 Oct 2019 06:28:43.060 * Before turning into a replica, using my master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
1:S 31 Oct 2019 06:28:43.060 * REPLICAOF 10.0.21.49:6379 enabled (user request from 'id=1085 addr=10.0.21.54:34360 fd=16 name=sentinel-592f3b97-cmd age=945 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=150 qbuf-free=32618 obl=36 oll=0 omem=0 events=r cmd=exec')
1:S 31 Oct 2019 06:28:43.701 * Connecting to MASTER 10.0.21.49:6379
1:S 31 Oct 2019 06:28:43.702 * MASTER <-> REPLICA sync started
1:S 31 Oct 2019 06:28:43.702 * Non blocking connect for SYNC fired the event.
1:S 31 Oct 2019 06:28:43.702 * Master replied to PING, replication can continue...
1:S 31 Oct 2019 06:28:43.703 * Trying a partial resynchronization (request a056665afb95a1e3a4227ae7fcb1c9b2e2f3b222:244418).
1:S 31 Oct 2019 06:28:43.703 * Full resync from master: adde2c9daee4fa1e62d3494d74d08dfb7110c798:241829
1:S 31 Oct 2019 06:28:43.703 * Discarding previously cached master state.
1:S 31 Oct 2019 06:28:43.715 * MASTER <-> REPLICA sync: receiving 2229 bytes from master
1:S 31 Oct 2019 06:28:43.715 * MASTER <-> REPLICA sync: Flushing old data
1:S 31 Oct 2019 06:28:43.715 * MASTER <-> REPLICA sync: Loading DB in memory
1:S 31 Oct 2019 06:28:43.715 * MASTER <-> REPLICA sync: Finished with success

MASTER MODE kicks in but then being taken over by REPLICAOF! How can I force redis-master to always be MASTER?

haytham
  • 502
  • 4
  • 22

1 Answers1

0

Yes, I think this makes sense.

Sentinel will always remember who joined the master-slave group.

When you manually make the current-slave master, sentinel didn't know if you make this on purpose, or there is a network portion that happened. So sentinels will do a convert-to-slave to avoid two masters exist in a group. (aka, split-brain)

To remove a node out of the group

Check the docs, In short, you need to send SENTINEL RESET mastername to all sentinels, to let them forget the lost node. Then start the lost node as master, it won't join the sentinel's group.

To make the previous (failed) master node stay as a master.

After the lost master coming back as a slave, you can do a SENTINEL failover <master name>, sentinels will do a failover and switch master and slave. But I don't think you can appoint a master when there are more than 3 nodes.

scriptboy
  • 777
  • 8
  • 25