1

We are trying to upgrade our redis/sentinel cluster from 2.8 to 3.2. Upgrade is going to happen under a heavy traffic. No downtime is acceptable for us.

Our setup has 6 redis/sentinels (3 sites, 2 servers on each site, each server has one instance of redis and sentinel running). Obviously we have 1 master and 5 slaves. We plan to upgrade our server one by one leaving our master server as the last one to upgrade.

By the time we are ready to upgrade our master, all slaves will be running on 3.2 and those slaves are able to sync up with master running on 2.8
Once we stop our master(2.8) or manually request a failover, a new server (3.2) will become master.

We want to keep our old master(2.8) around for a while, to be able to fallback in case if we find any issue with our new setup. Unfortunately an old redis(2.8) slave is not able to sync up with new master (3.2), because of different RDB format. We can stop our old server, slave(2.8), but we also want to be able to fallback to 2.8 as a master in case if we find an issue with 3.2. Since 2.8 is not able to sync up with 3.2, it would not have any data, thus can not be elected as new master by sentinel.

The question is how to fallback from 3.2 to 2.8 without losing any of the data?

pcapcanari
  • 11
  • 3
  • We managed to upgrade our cluster after running lot of test on a staging env. In the end there were no need to fallback after migration, since new cluster was stable during 2 weeks. We prepared a procedure for fallback. The procedure is leveraging redis AOF file. AOF file was taken from upgraded node and transferred to an old one. This was automated to minimize downtime and number of lost requests. – pcapcanari Jul 20 '17 at 14:09

1 Answers1

0

I wouldn't recommend keeping a 2.8 node around as a fallback. Like you mentioned, you can't sync 3.2 -> 2.8 so falling back to 2.8 will mean all writes made to 3.2 will be lost.

I would recommend setting up a staging environment that runs 3.2 and do whatever testing you need to do there. Once you are feeling relatively confident about it, take a backup of your production database and go through the migration process.

davissp14
  • 765
  • 4
  • 13