Form a new replica set with removed members

Question

How to configure removed members of a replica set to form new replica set?

I have a replica set with 4 mongod instances

Output of rs.config()

{
    "_id" : "rs0",
    "members" : [
        {
            "_id" : 0,
            "host" : "localhost:27031"
        },
        {
            "_id" : 1,
            "host" : "localhost:27032"
        },
        {
            "_id" : 2,
            "host" : "localhost:27033"
        },
        {
            "_id" : 3,
            "host" : "localhost:27034"
        }
    ],
    "settings" : {
       "replicaSetId" : ObjectId("5cf22332f5b9d21b01b9b6b2") 
    }
}

I removed 2 instances from the replica set

rs.remove("localhost:27033")
rs.remove("localhost:27034")

Now my requirement is to form a new replica set with these 2 removed members. What is the best way for that?

My current solution

connect to removed member

mongo --port 27033

and execute

conf = {
        "_id" : "rs0",
        "members" : [
            {
                "_id" : 2,
                "host" : "localhost:27033"
            },
            {
                "_id" : 3,
                "host" : "localhost:27034"
            }
        ],
        "settings" : {
           "replicaSetId" : ObjectId("5cf22332f5b9d21b01b9b6b2") 
        }
    }

and then

rs.reconfig(conf, {force:true})

Outcome

This solution worked fine practically. The removed members formed a replicaset, one of them became primary and other became secondary. Data was replicated among them. And this replica set seems to be isolated from the the initial replica set from which they were removed.

Concerns

1) I had to use forced reconfiguration. Not sure about the consequences.

"errmsg" : "replSetReconfig should only be run on PRIMARY, but my state is REMOVED; use the \"force\" argument to override",

2) Is the new replica set actually new one? In the rs.config()

replicaSetId is same as old one.

"replicaSetId" : ObjectId("5cf22332f5b9d21b01b9b6b2")
I had to use same value for _id of members as in config of old replica set

"errmsg" : "New and old configurations both have members with host of localhost:27034 but in the new configuration the _id field is 1 and in the old configuration it is 3 for replica set rs0",

Is this solution good?

Is there any better solution?

Note: I need to retain data from old replica set (data which was present at the time of removal) in the new replica set.

I think Mongo recommends odd number of replicaset members. – Nidhin David Jun 02 '19 at 03:04 — Nidhin David, Jun 02 '19 at 03:04
It is a good point. This is just a sample configuration. – Sreeragh A R Jun 02 '19 at 06:50 — Sreeragh A R, Jun 02 '19 at 06:50

score 7 · Accepted Answer · answered Jun 02 '19 at 23:48

As you have suspected, the procedure did not create a new replica set. Rather, it's a continuation of the old replica set, albeit superficially they look different.

There is actually a procedure in the MongoDB documentation to do what you want: Restore a Replica Set from MongoDB Backups. The difference being, you're not restoring from a backup. Rather, you're using one of the removed secondaries to seed a new replica set.

Hence you need to modify the first step in the procedure mentioned in the link above. The rest of the procedure would still be the same:

Restart the removed secondary as a standalone (without the --replSet parameter) and connect to it using the mongo shell.
Drop the local database in the standalone node:
```
use local
db.dropDatabase()
```
Restart the ex-secondary, this time with the --replSet parameter (with a new replica set name)
Connect to it using the mongo shell.
rs.initiate() the new set.

After this, the new set should have a different replicaSetId compared to the old set. In my quick test of the procedure above, this is the result I see:

Old set:

> rs.conf()
...
"replicaSetId": ObjectId("5cf45d72a1c6c4de948ff5d8")
...

New set

> rs.conf()
...
"replicaSetId": ObjectId("5cf45d000dda9e1025d6c65e")
...

As with any major deployment changes like this, please ensure that you have a backup, and thoroughly test the procedures before doing it on a production system.

Thanks for the answer. I also want to retain data in the removed members.(edited the question) Let me follow steps mentioned by you, avoiding "db.dropDatabase" and see. — Sreeragh A R, Jun 03 '19 at 05:13
The steps will preserve your data. The `local` database contains the necessary data for operating a replica set, so unless you have data inside `local` (which you shouldn't), it should work. — kevinadi, Jun 03 '19 at 05:53
Thanks for the clarification and solution. I think this solves my problem. — Sreeragh A R, Jun 03 '19 at 07:28

Form a new replica set with removed members

1 Answers1