replicaSet db status after failover and recover

Question

My mongodb version is 3.2.4. I have a replicaSet with 2 database nodes and 1 arbitor. All db are running fine for a long time at my customer site. One day, the primary db was brought down for maintenance. After about 2 hours, the-was-primary was brought back up, and becomes primary again, and secondary db is in Rollback state. I have a few questions regarding above mentioned scenarios: when the primary db was brought down the first time, if there are db entries haven't synced to the secondary db, what would happen? when the failed primary brought back up again, does it become primary right the way? Does it sync with now-primary-was-secondary db before becoming primary db again? how do I recover the lost data in rollback folder given my latest primary and secondary db state? Thanks and regards.

Answer 1

When you want to bring down a primary for maintenance, you'd have to do a rs.stepDown() command on the primary. This will elect the other DB node to become the primary:

• Primary steps down, it rejects writes. Your application will get brief write errors until the next bullet point below is completed.
• Secondary that gets elected will make sure it has synced up with the old primary before becoming a primary itself. This should happen in split-second, but if you have a write heavy application, it can take longer.
• When the old primary is brought back up, it will become a primary if you give it a highest priority. I would still recommend having equal priority for the 2 data nodes, and not return the primary back. The process of promoting the other node is exactly the same as the 2 bullet points above.

The rollback state you're in doesn't seem normal if you do a proper stepDown. Here's a good link to apply your rollback: https://scalegrid.io/blog/how-to-recover-from-a-mongodb-rollback/

Answer 2

Rick wrote good answer, but did not answer your last question... IF rollback happened, there will be rollback-directory under your dbpath. In that directory you can find all rollbacked documents in database-column specific json files. Those files can be mongoimport'ed back to the primary...