0

I have a three-node replica set (1 Primary, 1 Secondary, 1 Arbiter), on three different Amazon server instances. The servers where they are hosted required a memory upgrade so I needed to shut down the MongoDB instances as well.

I shut down the MongoDB instances in this order:

  1. Secondary
  2. Arbiter
  3. Primary

I used the process below for shutting down each server

use admin
db.shutdownServer()

All MongoDB instances did shut down properly without any problems. So far everything is fine.

After the Amazon server upgrade, I started the MongoDB instances in the following order:

  1. Arbiter
  2. Secondary
  3. Primary

The arbiter is in arbiter mode and secondary is in secondary mode, but to my surprise the primary machine went to "RECOVERING" mode.

I don't know the reason, why the primary machine went to "RECOVERING".

I have examined logs. It is showing no member to sync...something stuff like that

My basic doubt is "PRIMARY has to be in PRIMARY until there is reconfig happens in replica set".

Am I missing a step during the shutdown of servers? Or am I missing a step during the restart of servers?

Please shed some light on this so that how can I overcome this problem. I need to shut down the MongoDB servers frequently since there is a lot of upgrades happening in Amazon servers.

Vince Bowdren
  • 8,326
  • 3
  • 31
  • 56
Dhanu R
  • 1
  • 1

1 Answers1

0

After you started you replica set, you "SECONDARY" became "PRIMARY" and you "PRIMARY" was probably at secondary state after short while. To keep primary status at your "PRIMARY", you must give it higher priority than what your "SECONDARY" have.

Check with rs.conf() command.

Check here how to force node to be primary

JJussi
  • 1,540
  • 12
  • 12
  • after restart, primary went to recovering. Hence I cant make that back to primary – Dhanu R Dec 30 '16 at 04:37
  • So, it went to recovering and stayed.. Strange. Do, full initial sync. Shutdown that node, delete all files from data dir and start node again, it will resync from other node. (of course before that you could check log-file what it says about reason to go to recovering state) – JJussi Dec 31 '16 at 16:37