1

Consider you have to move your mongo cluster from one environment to another, with minimal downtime during the process.

Let's say you have environment 1 (i.e AWS), with the following mongo nodes:

  • mongo_aws_1
  • mongo_aws_2
  • mongo_aws_3

Your application is configured to talk to mongo_aws_1, 2, 3, with hardcoded ip addresses and no DNS names.

What would be the easiest way to move the cluster to another environment (i.e Google Cloud), so that you have

  • mongo_gc_1
  • mongo_gc_2
  • mongo_gc_3 ?

My migration plan was the following (this ofc assumes connectivity between AWS and GC)

  • create new mongo nodes (mongo_gc_1, 2, 3) and add them to existing cluster, with priority = 0 (rs.add('host': 'mongo_gc_1:27017', 'priority': 0)). Then wait until the nodes synchronize with replicaset (the entire dataset is less than 15GB, so it's ok to wait)
  • redeploy the application so that it talks to both AWS and GC nodes
  • change the priority of GC nodes to 1 so that they become fully-fledged members (rs.reconfig())
  • remove 'old secondaries' from the cluster
  • ask PRIMARY to stepdown (this will force one of GC nodes to become primary)
  • remove 'old primary' from the cluster
  • redeploy application so it only talks to GC nodes

This should work and the only downtime it introduces is when the old primary steps down.

However, it assumes that primary does not change during the entire operation. On the other hand, according to documentation:

The rs.reconfig() shell method can force the current primary to step down, which causes an election

also, when you remove a node from the replicaset:

This function will disconnect the shell briefly and forces a reconnection as the replica set renegotiates which member will be primary. As a result, the shell will display an error even if this command succeeds.

What should be the proper way of implementing this? Can I somehow avoid the unncessary reelections?

slnowak
  • 1,839
  • 3
  • 23
  • 37

1 Answers1

0

When you add your 3 GC nodes you could also make them hidden. Not only can hidden members not become primary they are invisible to client applications.

rs.add() provides a wrapper around some of the functionality of the replSetReconfig db command and the corresponding shell helper rs.reconfig:

replSetReconfig: This command can cause downtime as the set renegotiates primary-status. Therefore, you should attempt to reconfigure only during scheduled maintenance periods.

So by doing it the way you've proposed could cause 3 separate elections: 

rs.add({host: "mongo_gc_1:27017", priority: 0 })
rs.add({host: "mongo_gc_2:27017", priority: 0 })
rs.add({host: "mongo_gc_3:27017", priority: 0 })

You could get that possibility down to only 1 by combining you priority and/or hidden change in one reconfig call:

rs.add({host: "mongo_gc_1:27017" })
rs.add({host: "mongo_gc_2:27017" })
rs.add({host: "mongo_gc_3:27017" })

conf = rs.conf() 

conf.members[4].priority = 0
conf.members[5].priority = 0
conf.members[6].priority = 0

conf.members[4].hidden = true
conf.members[5].hidden = true
conf.members[6].hidden = true

rs.reconfig(conf)
  • A few points here: - it is possible to add node with priority 0, I've already tested it - it is probably not possible to lower the priority of existing PRIMARY to 0 (keep getting NewReplicaSetConfigurationIncompatible) Is it true that rs.reconfig only triggers election only in case the priority of some other node is higher that the priority of current primary? – slnowak Feb 20 '18 at 09:04
  • @slnowak I've updated my answer regarding the `rs.add`. The documentation is rather vague regarding `rs.reconfig` "can trigger the current primary to step down in some situations" what "some situations' means will probably require testing to find out –  Feb 20 '18 at 09:44