1

I just installed MongoDB on macOS and I was writing billions of documents on it by multiple threads (for a realistic benchmark with our own real data - each document has around 500 bytes in average containing integer, float and binary data fields). After writing 10-20 millions of documents, suddenly MongoDB service stopped working.

brew services restart mongodb just starts mongodb and it automatically stops after ~10 seconds. Activity Monitor says it writes around 700mb of data to the disk during this 10 seconds and reads almost same amount of data.

mongo command fails with following output:

$ mongo
MongoDB shell version v3.6.2
connecting to: mongodb://127.0.0.1:27017
***TIME*** W NETWORK  [thread1] Failed to connect to 127.0.0.1:27017, in(checking socket for error after poll), reason: Connection refused
***TIME*** E QUERY    [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
connect@src/mongo/shell/mongo.js:251:13
@(connect):1:6
exception: connect failed

Here is the log file when I try brew services restart mongodb:

***TIME*** I CONTROL  [main] ***** SERVER RESTARTED *****
***TIME*** I CONTROL  [initandlisten] MongoDB starting : pid=58235 port=27017 dbpath=/usr/local/var/mongodb 64-bit host=***HOST***
***TIME*** I CONTROL  [initandlisten] db version v3.6.2
***TIME*** I CONTROL  [initandlisten] git version: 489d177dbd0f0420a8ca04d39fd78d0a2c539420
***TIME*** I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2o  27 Mar 2018
***TIME*** I CONTROL  [initandlisten] allocator: system
***TIME*** I CONTROL  [initandlisten] modules: none
***TIME*** I CONTROL  [initandlisten] build environment:
***TIME*** I CONTROL  [initandlisten]     distarch: x86_64
***TIME*** I CONTROL  [initandlisten]     target_arch: x86_64
***TIME*** I CONTROL  [initandlisten] options: { config: "/usr/local/etc/mongod.conf", net: { bindIp: "127.0.0.1" }, storage: { dbPath: "/usr/local/var/mongodb" }, systemLog: { destination: "file", logAppend: true, path: "/logs/mongodb.log" } }
***TIME*** I -        [initandlisten] Detected data files in /usr/local/var/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
***TIME*** I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1536M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
***TIME*** I STORAGE  [initandlisten] WiredTiger message [1524603520:902878][58235:0x7fffc82c43c0], txn-recover: Main recovery loop: starting at 91/768
***TIME*** I STORAGE  [initandlisten] WiredTiger message [1524603521:64660][58235:0x7fffc82c43c0], txn-recover: Recovering log 91 through 92
***TIME*** I STORAGE  [initandlisten] WiredTiger message [1524603521:280503][58235:0x7fffc82c43c0], txn-recover: Recovering log 92 through 92
***TIME*** I CONTROL  [initandlisten] 
***TIME*** I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
***TIME*** I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
***TIME*** I CONTROL  [initandlisten] 
***TIME*** F CONTROL  [initandlisten] ** IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to version 3.4 before attempting an upgrade to 3.6; see http://dochub.mongodb.org/core/3.6-upgrade-fcv for more details.
***TIME*** I NETWORK  [initandlisten] shutdown: going to close listening sockets...
***TIME*** I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock
***TIME*** I REPL     [initandlisten] shutdown: removing all drop-pending collections...
***TIME*** I REPL     [initandlisten] shutdown: removing checkpointTimestamp collection...
***TIME*** I STORAGE  [initandlisten] WiredTigerKVEngine shutting down
***TIME*** I STORAGE  [initandlisten] WiredTiger message [1524603524:5088][58235:0x7fffc82c43c0], txn-recover: Main recovery loop: starting at 92/3072
***TIME*** I STORAGE  [initandlisten] WiredTiger message [1524603524:182001][58235:0x7fffc82c43c0], txn-recover: Recovering log 92 through 93
***TIME*** I STORAGE  [initandlisten] WiredTiger message [1524603524:408698][58235:0x7fffc82c43c0], txn-recover: Recovering log 93 through 93
***TIME*** I STORAGE  [initandlisten] shutdown: removing fs lock...
***TIME*** I CONTROL  [initandlisten] now exiting
***TIME*** I CONTROL  [initandlisten] shutting down with code:62

How can I fix the issue and start mongodb service again?

Stennie
  • 63,885
  • 14
  • 149
  • 175
S.D
  • 29
  • 4
  • `mongo` is the mongo shell. Since the service isn't running, it cannot connect to the mongod service. Try running `mongod` from the command line to see what errors the database produces after 10 seconds when it fails and shuts down. Alternatively, look at the log, maybe located @ `/usr/local/var/log/mongodb/mongo.log` (https://stackoverflow.com/q/27564415/2721883) – clcto Apr 24 '18 at 20:55
  • @clcto I just appended log file to the question – S.D Apr 24 '18 at 21:06
  • 2
    Hmmn. `** IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to version 3.4 before attempting an upgrade to 3.6; see http://dochub.mongodb.org/core/3.6-upgrade-fcv for more details.` Helps to read your log files and the links in the documentation they provide. Noting that in your question you failed to tell people you were actually attempting an install on top of a previous version. Details like that matter. Read the link in the logs and follow the instructions. – Neil Lunn Apr 24 '18 at 22:07
  • It looks like you've upgraded from an older version of MongoDB and skipped one or more major releases in the process (eg. 3.2 => 3.6). The supported upgrade path is currently through consecutive major releases, so an upgrade from 3.2 to 3.6 would involve 3.2 => 3.4 followed by 3.4 => 3.6. Upgrade prerequisites, compatibility notes, and procedures are covered in the [MongoDB release notes](https://docs.mongodb.com/manual/release-notes/). – Stennie Apr 24 '18 at 23:58
  • I'm sure I havn't upgraded, but I will check it again. Thanks. – S.D Apr 25 '18 at 01:37

0 Answers0