-1

we are trying to move from mongo 2.4.9 to 3.4, we have a lot of data so we tried to set replication and wait while data will be synced and then swap primary. Configurations done but when replication is initiated new server cant stabilize replication:

017-07-07T12:07:22.492+0000 I REPL     [replication-1] Starting initial sync (attempt 10 of 10)
2017-07-07T12:07:22.501+0000 I REPL     [replication-1] sync source candidate: mongo-2.blabla.com:27017
2017-07-07T12:07:22.501+0000 I STORAGE  [replication-1] dropAllDatabasesExceptLocal 1
2017-07-07T12:07:22.501+0000 I REPL     [replication-1] ******
2017-07-07T12:07:22.501+0000 I REPL     [replication-1] creating replication oplog of size: 6548MB...
2017-07-07T12:07:22.504+0000 I STORAGE  [replication-1] WiredTigerRecordStoreThread local.oplog.rs already started
2017-07-07T12:07:22.505+0000 I STORAGE  [replication-1] The size storer reports that the oplog contains 0 records totaling to 0 bytes
2017-07-07T12:07:22.505+0000 I STORAGE  [replication-1] Scanning the oplog to determine where to place markers for truncation
2017-07-07T12:07:22.519+0000 I REPL     [replication-1] ******
2017-07-07T12:07:22.521+0000 I REPL     [replication-1] Initial sync attempt finishing up.
2017-07-07T12:07:22.521+0000 I REPL     [replication-1] Initial Sync Attempt Statistics: { failedInitialSyncAttempts: 9, maxFailedInitialSyncAttempts: 10, initialSyncStart: new Date(1499429233163), initialSyncAttempts: [ { durationMillis: 0, status: "CommandNotFound: error while getting last oplog entry for begin timestamp: no such cmd: find", syncSource: "mongo-2.blabla.com:27017" }, { durationMillis: 0, status: "CommandNotFound: error while getting last oplog entry for begin timestamp: no such cmd: find", syncSource: "mongo-2.blabla.com:27017" }, { durationMillis: 0, status: "CommandNotFound: error while getting last oplog entry for begin timestamp: no such cmd: find", syncSource: "mongo-2.blabla.com:27017" }, { durationMillis: 0, status: "CommandNotFound: error while getting last oplog entry for begin timestamp: no such cmd: find", syncSource: "mongo-2.blabla.com:27017" }, { durationMillis: 0, status: "CommandNotFound: error while getting last oplog entry for begin timestamp: no such cmd: find", syncSource: "mongo-2.blabla.com:27017" }, { durationMillis: 0, status: "CommandNotFound: error while getting last oplog entry for begin timestamp: no such cmd: find", syncSource: "mongo-2.blabla.com:27017" }, { durationMillis: 0, status: "CommandNotFound: error while getting last oplog entry for begin timestamp: no such cmd: find", syncSource: "mongo-2.blabla.com:27017" }, { durationMillis: 0, status: "CommandNotFound: error while getting last oplog entry for begin timestamp: no such cmd: find", syncSource: "mongo-2.blabla.com:27017" }, { durationMillis: 0, status: "CommandNotFound: error while getting last oplog entry for begin timestamp: no such cmd: find", syncSource: "mongo-2.blabla.com:27017" } ] }
2017-07-07T12:07:22.521+0000 E REPL     [replication-1] Initial sync attempt 
failed -- attempts left: 0 cause: CommandNotFound: error while getting last 
oplog entry for begin timestamp: no such cmd: find
2017-07-07T12:07:22.521+0000 F REPL     [replication-1] The maximum number 
of retries have been exhausted for initial sync.
2017-07-07T12:07:22.522+0000 E REPL     [replication-0] Initial sync failed, 

shutting down now. Restart the server to attempt a new initial sync.
2017-07-07T12:07:22.522+0000 I -        [replication-0] Fatal assertion 40088 CommandNotFound: error while getting last oplog entry for begin timestamp: no such cmd: find at src/mongo/db/repl/replication_coordinator_impl.cpp 632

please assits guys, since we have more than 100G of data, so dump and restore will take a lot of downtime

Configurations: 3.4.5 new machine: 

storage: 
  dbPath: /mnt/dbpath 
  journal: 
    enabled: true 
  engine: wiredTiger 
  systemLog: 
    destination: file
    logAppend: true 
    path: /var/log/mongodb/mongod.log 
  net: 
    port: 27017 
  replication: 
    replSetName: prodTest

2.4.9 old machine with data: 

dbpath=/var/lib/mongodb 
logpath=/var/log/mongodb/mongodb.log 
logappend=true port = 27017
Alex Sharabudinov
  • 1
  • 1
  • post your mongodb config options – Ratan Uday Kumar Jul 07 '17 at 15:59
  • **3.4.5 new machine:** storage: dbPath: /mnt/dbpath journal: enabled: true engine: wiredTiger systemLog: destination: file logAppend: true path: /var/log/mongodb/mongod.log net: port: 27017 replication: replSetName: prodTest **2.4.9 old machine with data:** dbpath=/var/lib/mongodb logpath=/var/log/mongodb/mongodb.log logappend=true port = 27017 – Alex Sharabudinov Jul 07 '17 at 16:02
  • It looks like you're trying to setup a mixed replica set using MongoDB 2.4.x and 3.4.x in order to upgrade. That won't work: mixed replication for the purposes of upgrades is only supported between adjacent major versions (2.4 => 2.6, 2.6 => 3.0). For supported approaches to successfully upgrade please review the [MongoDB Release Notes](https://docs.mongodb.com/manual/release-notes/). Questions on upgrade or admin are also more on-topic for [DBA StackExchange](https://dba.stackexchange.com/tags/mongodb). – Stennie Jul 07 '17 at 16:04
  • Thanks @Stennie is it possible to do chaining replication: **2.4.9->2.6->3.4.5**? Since mongodump cant be restored on new mongoversion as well( – Alex Sharabudinov Jul 07 '17 at 16:09
  • 1
    You can't mix more than two major versions in a single replica set. You should be able to use `mongodump`, although given the number of versions you are skipping there are some caveats on checking for index compatibility to ensure a smooth upgrade. It also looks like you aren't using auth, which simplifies some upgrade steps since the auth schema has evolved significantly since 2.4.x. I suggest removing this StackOverflow question & posting a new one on DBA StackExchange outlining your current deployment details and upgrade requirements. For example, how much downtime can you tolerate? – Stennie Jul 07 '17 at 16:21
  • @Stennie thanks. So basically i can do such plan: 1 replica between 2.4.9 and 2.6.x (swap master after sync finish) 2 replica between 2.6 and 3.0 (swap master after sync finish) 4 replica between 3.0 and 3.4 (swap master after sync finish) Each swap will require downtime in order to give few minutes for last oplog apply. We are eligible for downtime less than 30m only. – Alex Sharabudinov Jul 10 '17 at 09:50
  • I was actually suggesting posting a question with more detail on DBA StackExchange .. I don't think you are ready for a plan yet ;-). So far I know that you are looking for a shortcut to jump from 2.4 to 3.4 (aka four major version upgrades) but haven't read the upgrade notes in the documentation yet (where prerequisites and caveats are noted). You also missed a step in upgrades (3.0 => 3.2, 3.2 => 3.4). With rolling upgrades done correctly you should have minimal downtime but that's a lot of versions and you will also need to update your drivers (particularly if you plan on using WiredTiger). – Stennie Jul 10 '17 at 15:21
  • @Stennie You should post your comments as an answer, they deserve an upvote. – Vince Bowdren Jul 11 '17 at 12:38

1 Answers1

0

the task have been solved in such way:

  1. -create replica master-v2.4, 3 slaves-v2.6
  2. -stop app, step down master
  3. -stop new master and upgrade mongo version to v3.0,
  4. start master and upgrade slaves sequentually to 3.2(slave db files removed new version started on wiredTiger engine)
  5. -step down master, upgrade all slaves to 3.4

This process become very fast because replica slave recovery of 40G db takes around 30m.

Alex Sharabudinov
  • 1
  • 1