0

I am working on a project, In which we are processing high amount of data. We are using mongoDb as database into our project, I have setup replicas on my mongoDb, with 3 replica sets(1 as primary, 2 secondary) and my system configuration is:- RAM:- 128GB(In each system) OS:- ubuntu 22.04 monngoDb version:- 4.4.1

My problem is, whenever i have started the mongoDb server, for sometime it was working fine, and after a few time, it becomes slow, it starts taking pause in between the processes, And time of these pause are increasing rapidly, e.g. when first time mongoDb stops, it will start in 1-5 seconds, and this time will increase rapidly, after some time it will take 20s to start whenever it was stops, and this time will goes to 2-3 minutes during the end of the day. And I am facing this mongodb stop issue in every 1-2 minutes.

While looking at the mongoDb logs, I have found some suspicious log that was logging continously, whenever the db stops working.

{"t":{"$date":"2023-03-16T12:39:32.469+00:00"},"s":"D1", "c":"REPL",     "id":21342,   "ctx":"conn9","msg":"Waiting for an isMaster response from a topology change or until deadline","attr":{"deadline":{"$date":"2023-03-16T12:39:42.469Z"},"currentTopologyVersionCounter":4}}
{"t":{"$date":"2023-03-16T12:39:21.554+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn33364","msg":"Slow query","attr":{"type":"command","ns":"admin.$cmd","command":{"hello":1,"helloOk":true,"topologyVersion":{"processId":{"$oid":"6412883cae50b6d202b8c8bc"},"counter":4},"maxAwaitTimeMS":10000,"$db":"admin","$readPreference":{"mode":"primaryPreferred"}},"numYields":0,"reslen":788,"locks":{},"protocol":"op_msg","durationMillis":0}}

Please help me out with this issue. FYI, there was around 10k/s read operations and around 4-5k/s update and insert operations..

here are some more logs that I am facing. primary Db logs:-

{"t":{"$date":"2023-03-17T10:52:42.139+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn37","msg":"Slow query","attr":{"type":"command","ns":"local.oplog.rs","appName":"OplogFetcher","command":{"getMore":6225246966303344778,"collection":"oplog.rs","batchSize":13981010,"maxTimeMS":5000,"term":98,"lastKnownCommittedOpTime":{"ts":{"$timestamp":{"t":0,"i":0}},"t":-1},"$db":"local","$replData":1,"$oplogQueryData":1,"$readPreference":{"mode":"secondaryPreferred"},"$clusterTime":{"clusterTime":{"$timestamp":{"t":1679035194,"i":1}},"signature":{"hash":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAAAAAAA=","subType":"0"}},"keyId":0}}},"originatingCommand":{"find":"oplog.rs","filter":{"ts":{"$gte":{"$timestamp":{"t":1679035184,"i":1}}}},"readConcern":{"level":"local","afterClusterTime":{"$timestamp":{"t":0,"i":1}}},"ntoreturn":13981010,"maxTimeMS":60000,"tailable":true,"awaitData":true,"oplogReplay":true,"term":98,"$db":"local","$replData":1,"$oplogQueryData":1,"$readPreference":{"mode":"secondaryPreferred"},"$clusterTime":{"clusterTime":{"$timestamp":{"t":1679035194,"i":1}},"signature":{"hash":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAAAAAAA=","subType":"0"}},"keyId":0}}},"planSummary":"COLLSCAN","cursorid":6225246966303344778,"keysExamined":0,"docsExamined":3,"numYields":2,"nreturned":3,"reslen":2772,"locks":{"FeatureCompatibilityVersion":{"acquireCount":{"r":3}},"ReplicationStateTransition":{"acquireCount":{"w":3}},"Global":{"acquireCount":{"r":3}},"Database":{"acquireCount":{"r":3}},"Mutex":{"acquireCount":{"r":1}},"oplog":{"acquireCount":{"r":3}}},"readConcern":{"level":"local","afterClusterTime":{"$timestamp":{"t":0,"i":1}},"provenance":"clientSupplied"},"storage":{},"protocol":"op_msg","durationMillis":1}}
{"t":{"$date":"2023-03-17T10:52:42.139+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn32147","msg":"Slow query","attr":{"type":"command","ns":"local.oplog.rs","appName":"OplogFetcher","command":{"getMore":5729678527266085484,"collection":"oplog.rs","batchSize":13981010,"maxTimeMS":5000,"term":99,"lastKnownCommittedOpTime":{"ts":{"$timestamp":{"t":1679049883,"i":10}},"t":99},"$db":"local","$replData":1,"$oplogQueryData":1,"$readPreference":{"mode":"secondaryPreferred"},"$clusterTime":{"clusterTime":{"$timestamp":{"t":1679049884,"i":87}},"signature":{"hash":{"$binary":{"base64":"FYO1bojgbetft/tif+Wt2TpU+Q8=","subType":"0"}},"keyId":7205967400469004293}}},"originatingCommand":{"find":"oplog.rs","filter":{"ts":{"$gte":{"$timestamp":{"t":1679049837,"i":98}}}},"readConcern":{"level":"local","afterClusterTime":{"$timestamp":{"t":0,"i":1}}},"ntoreturn":13981010,"maxTimeMS":60000,"tailable":true,"awaitData":true,"oplogReplay":true,"term":99,"$db":"local","$replData":1,"$oplogQueryData":1,"$readPreference":{"mode":"secondaryPreferred"},"$clusterTime":{"clusterTime":{"$timestamp":{"t":1679049884,"i":1}},"signature":{"hash":{"$binary":{"base64":"FYO1bojgbetft/tif+Wt2TpU+Q8=","subType":"0"}},"keyId":7205967400469004293}}},"planSummary":"COLLSCAN","cursorid":5729678527266085484,"keysExamined":0,"docsExamined":3,"numYields":2,"nreturned":3,"reslen":2772,"locks":{"FeatureCompatibilityVersion":{"acquireCount":{"r":3}},"ReplicationStateTransition":{"acquireCount":{"w":3}},"Global":{"acquireCount":{"r":3}},"Database":{"acquireCount":{"r":3}},"Mutex":{"acquireCount":{"r":1}},"oplog":{"acquireCount":{"r":3}}},"readConcern":{"level":"local","afterClusterTime":{"$timestamp":{"t":0,"i":1}},"provenance":"clientSupplied"},"storage":{},"protocol":"op_msg","durationMillis":1}}
{"t":{"$date":"2023-03-17T10:52:42.140+00:00"},"s":"D1", "c":"STORAGE",  "id":22411,   "ctx":"conn32048","msg":"Slow WT transaction. Lifetime of SnapshotId {getSnapshotId_toNumber} was {transactionTime}ms","attr":{"getSnapshotId_toNumber":163045121,"transactionTime":1400}}
{"t":{"$date":"2023-03-17T10:52:42.140+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn37","msg":"Slow query","attr":{"type":"command","ns":"local.oplog.rs","appName":"OplogFetcher","command":{"getMore":6225246966303344778,"collection":"oplog.rs","batchSize":13981010,"maxTimeMS":5000,"term":98,"lastKnownCommittedOpTime":{"ts":{"$timestamp":{"t":0,"i":0}},"t":-1},"$db":"local","$replData":1,"$oplogQueryData":1,"$readPreference":{"mode":"secondaryPreferred"},"$clusterTime":{"clusterTime":{"$timestamp":{"t":1679035194,"i":1}},"signature":{"hash":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAAAAAAA=","subType":"0"}},"keyId":0}}},"originatingCommand":{"find":"oplog.rs","filter":{"ts":{"$gte":{"$timestamp":{"t":1679035184,"i":1}}}},"readConcern":{"level":"local","afterClusterTime":{"$timestamp":{"t":0,"i":1}}},"ntoreturn":13981010,"maxTimeMS":60000,"tailable":true,"awaitData":true,"oplogReplay":true,"term":98,"$db":"local","$replData":1,"$oplogQueryData":1,"$readPreference":{"mode":"secondaryPreferred"},"$clusterTime":{"clusterTime":{"$timestamp":{"t":1679035194,"i":1}},"signature":{"hash":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAAAAAAA=","subType":"0"}},"keyId":0}}},"planSummary":"COLLSCAN","cursorid":6225246966303344778,"keysExamined":0,"docsExamined":1,"numYields":2,"nreturned":1,"reslen":1398,"locks":{"FeatureCompatibilityVersion":{"acquireCount":{"r":3}},"ReplicationStateTransition":{"acquireCount":{"w":3}},"Global":{"acquireCount":{"r":3}},"Database":{"acquireCount":{"r":3}},"Mutex":{"acquireCount":{"r":1}},"oplog":{"acquireCount":{"r":3}}},"readConcern":{"level":"local","afterClusterTime":{"$timestamp":{"t":0,"i":1}},"provenance":"clientSupplied"},"storage":{},"protocol":"op_msg","durationMillis":0}}
{"t":{"$date":"2023-03-17T10:52:42.140+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn32147","msg":"Slow query","attr":{"type":"command","ns":"local.oplog.rs","appName":"OplogFetcher","command":{"getMore":5729678527266085484,"collection":"oplog.rs","batchSize":13981010,"maxTimeMS":5000,"term":99,"lastKnownCommittedOpTime":{"ts":{"$timestamp":{"t":1679049883,"i":10}},"t":99},"$db":"local","$replData":1,"$oplogQueryData":1,"$readPreference":{"mode":"secondaryPreferred"},"$clusterTime":{"clusterTime":{"$timestamp":{"t":1679049884,"i":87}},"signature":{"hash":{"$binary":{"base64":"FYO1bojgbetft/tif+Wt2TpU+Q8=","subType":"0"}},"keyId":7205967400469004293}}},"originatingCommand":{"find":"oplog.rs","filter":{"ts":{"$gte":{"$timestamp":{"t":1679049837,"i":98}}}},"readConcern":{"level":"local","afterClusterTime":{"$timestamp":{"t":0,"i":1}}},"ntoreturn":13981010,"maxTimeMS":60000,"tailable":true,"awaitData":true,"oplogReplay":true,"term":99,"$db":"local","$replData":1,"$oplogQueryData":1,"$readPreference":{"mode":"secondaryPreferred"},"$clusterTime":{"clusterTime":{"$timestamp":{"t":1679049884,"i":1}},"signature":{"hash":{"$binary":{"base64":"FYO1bojgbetft/tif+Wt2TpU+Q8=","subType":"0"}},"keyId":7205967400469004293}}},"planSummary":"COLLSCAN","cursorid":5729678527266085484,"keysExamined":0,"docsExamined":1,"numYields":2,"nreturned":1,"reslen":1398,"locks":{"FeatureCompatibilityVersion":{"acquireCount":{"r":3}},"ReplicationStateTransition":{"acquireCount":{"w":3}},"Global":{"acquireCount":{"r":3}},"Database":{"acquireCount":{"r":3}},"Mutex":{"acquireCount":{"r":1}},"oplog":{"acquireCount":{"r":3}}},"readConcern":{"level":"local","afterClusterTime":{"$timestamp":{"t":0,"i":1}},"provenance":"clientSupplied"},"storage":{},"protocol":"op_msg","durationMillis":0}}
{"t":{"$date":"2023-03-17T10:52:42.142+00:00"},"s":"D1", "c":"REPL",     "id":21337,   "ctx":"JournalFlusher","msg":"Required snapshot optime is not yet part of the current 'committed' snapshot","attr":{"opTime":{"ts":{"$timestamp":{"t":1679050359,"i":9}},"t":99},"currentCommittedSnapshotOpTime":{"ts":{"$timestamp":{"t":1679050359,"i":8}},"t":99}}}
{"t":{"$date":"2023-03-17T10:52:42.143+00:00"},"s":"D1", "c":"REPL",     "id":21337,   "ctx":"JournalFlusher","msg":"Required snapshot optime is not yet part of the current 'committed' snapshot","attr":{"opTime":{"ts":{"$timestamp":{"t":1679050359,"i":10}},"t":99},"currentCommittedSnapshotOpTime":{"ts":{"$timestamp":{"t":1679050359,"i":8}},"t":99}}}
{"t":{"$date":"2023-03-17T10:52:42.143+00:00"},"s":"D1", "c":"REPL",     "id":21337,   "ctx":"JournalFlusher","msg":"Required snapshot optime is not yet part of the current 'committed' snapshot","attr":{"opTime":{"ts":{"$timestamp":{"t":1679050359,"i":11}},"t":99},"currentCommittedSnapshotOpTime":{"ts":{"$timestamp":{"t":1679050359,"i":8}},"t":99}}}
{"t":{"$date":"2023-03-17T10:52:42.143+00:00"},"s":"D1", "c":"REPL",     "id":21337,   "ctx":"JournalFlusher","msg":"Required snapshot optime is not yet part of the current 'committed' snapshot","attr":{"opTime":{"ts":{"$timestamp":{"t":1679050359,"i":12}},"t":99},"currentCommittedSnapshotOpTime":{"ts":{"$timestamp":{"t":1679050359,"i":8}},"t":99}}}
{"t":{"$date":"2023-03-17T10:52:42.143+00:00"},"s":"D1", "c":"REPL",     "id":21337,   "ctx":"JournalFlusher","msg":"Required snapshot optime is not yet part of the current 'committed' snapshot","attr":{"opTime":{"ts":{"$timestamp":{"t":1679050359,"i":13}},"t":99},"currentCommittedSnapshotOpTime":{"ts":{"$timestamp":{"t":1679050359,"i":8}},"t":99}}}
{"t":{"$date":"2023-03-17T10:52:42.143+00:00"},"s":"D1", "c":"REPL",     "id":21337,   "ctx":"JournalFlusher","msg":"Required snapshot optime is not yet part of the current 'committed' snapshot","attr":{"opTime":{"ts":{"$timestamp":{"t":1679050359,"i":14}},"t":99},"currentCommittedSnapshotOpTime":{"ts":{"$timestamp":{"t":1679050359,"i":8}},"t":99}}}
{"t":{"$date":"2023-03-17T10:52:42.143+00:00"},"s":"D1", "c":"REPL",     "id":21337,   "ctx":"JournalFlusher","msg":"Required snapshot optime is not yet part of the current 'committed' snapshot","attr":{"opTime":{"ts":{"$timestamp":{"t":1679050359,"i":15}},"t":99},"currentCommittedSnapshotOpTime":{"ts":{"$timestamp":{"t":1679050359,"i":8}},"t":99}}}
{"t":{"$date":"2023-03-17T10:52:42.143+00:00"},"s":"D1", "c":"REPL",     "id":21337,   "ctx":"JournalFlusher","msg":"Required snapshot optime is not yet part of the current 'committed' snapshot","attr":{"opTime":{"ts":{"$timestamp":{"t":1679050359,"i":16}},"t":99},"currentCommittedSnapshotOpTime":{"ts":{"$timestamp":{"t":1679050359,"i":8}},"t":99}}}
{"t":{"$date":"2023-03-17T10:52:42.143+00:00"},"s":"D1", "c":"REPL",     "id":21337,   "ctx":"JournalFlusher","msg":"Required snapshot optime is not yet part of the current 'committed' snapshot","attr":{"opTime":{"ts":{"$timestamp":{"t":1679050359,"i":19}},"t":99},"currentCommittedSnapshotOpTime":{"ts":{"$timestamp":{"t":1679050359,"i":8}},"t":99}}}
{"t":{"$date":"2023-03-17T10:52:42.143+00:00"},"s":"D1", "c":"REPL",     "id":21337,   "ctx":"JournalFlusher","msg":"Required snapshot optime is not yet part of the current 'committed' snapshot","attr":{"opTime":{"ts":{"$timestamp":{"t":1679050359,"i":22}},"t":99},"currentCommittedSnapshotOpTime":{"ts":{"$timestamp":{"t":1679050359,"i":8}},"t":99}}}
{"t":{"$date":"2023-03-17T10:52:42.143+00:00"},"s":"D1", "c":"REPL",     "id":21337,   "ctx":"JournalFlusher","msg":"Required snapshot optime is not yet part of the current 'committed' snapshot","attr":{"opTime":{"ts":{"$timestamp":{"t":1679050359,"i":26}},"t":99},"currentCommittedSnapshotOpTime":{"ts":{"$timestamp":{"t":1679050359,"i":8}},"t":99}}}
{"t":{"$date":"2023-03-17T10:52:42.143+00:00"},"s":"D1", "c":"REPL",     "id":21337,   "ctx":"JournalFlusher","msg":"Required snapshot optime is not yet part of the current 'committed' snapshot","attr":{"opTime":{"ts":{"$timestamp":{"t":1679050359,"i":27}},"t":99},"currentCommittedSnapshotOpTime":{"ts":{"$timestamp":{"t":1679050359,"i":8}},"t":99}}}
{"t":{"$date":"2023-03-17T10:52:42.143+00:00"},"s":"D1", "c":"REPL",     "id":21337,   "ctx":"JournalFlusher","msg":"Required snapshot optime is not yet part of the current 'committed' snapshot","attr":{"opTime":{"ts":{"$timestamp":{"t":1679050359,"i":28}},"t":99},"currentCommittedSnapshotOpTime":{"ts":{"$timestamp":{"t":1679050359,"i":8}},"t":99}}}
{"t":{"$date":"2023-03-17T10:52:42.143+00:00"},"s":"D1", "c":"REPL",     "id":21337,   "ctx":"JournalFlusher","msg":"Required snapshot optime is not yet part of the current 'committed' snapshot","attr":{"opTime":{"ts":{"$timestamp":{"t":1679050359,"i":29}},"t":99},"currentCommittedSnapshotOpTime":{"ts":{"$timestamp":{"t":1679050359,"i":8}},"t":99}}}

secondary db logs:-

{"t":{"$date":"2023-03-17T10:47:21.176+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn14273","msg":"Slow query","attr":{"type":"command","ns":"admin.$cmd","command":{"hello":1,"helloOk":true,"$clusterTime":{"clusterTime":{"$timestamp":{"t":1679050041,"i":1}},"signature":{"hash":{"$binary":{"base64":"57Ph/I8TFtaxZoDJzNMB6rD2Meo=","subType":"0"}},"keyId":7205967400469004293}},"$db":"admin","$readPreference":{"mode":"primaryPreferred"}},"numYields":0,"reslen":788,"locks":{},"protocol":"op_msg","durationMillis":0}}
{"t":{"$date":"2023-03-17T10:47:21.298+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn14274","msg":"Slow query","attr":{"type":"command","ns":"admin.$cmd","command":{"hello":1,"helloOk":true,"topologyVersion":{"processId":{"$oid":"641410c417541421e074328b"},"counter":4},"maxAwaitTimeMS":10000,"$db":"admin","$readPreference":{"mode":"primaryPreferred"}},"numYields":0,"reslen":788,"locks":{},"protocol":"op_msg","durationMillis":0}}
{"t":{"$date":"2023-03-17T10:47:21.298+00:00"},"s":"D1", "c":"REPL",     "id":21342,   "ctx":"conn14274","msg":"Waiting for an isMaster response from a topology change or until deadline","attr":{"deadline":{"$date":"2023-03-17T10:47:31.298Z"},"currentTopologyVersionCounter":4}}
{"t":{"$date":"2023-03-17T10:47:21.364+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn14","msg":"Slow query","attr":{"type":"command","ns":"admin.$cmd","command":{"replSetHeartbeat":"stgmaindb","configVersion":3,"configTerm":99,"hbv":1,"from":ip,"fromId":0,"term":99,"primaryId":0,"$replData":1,"$clusterTime":{"clusterTime":{"$timestamp":{"t":1679050041,"i":1}},"signature":{"hash":{"$binary":{"base64":"57Ph/I8TFtaxZoDJzNMB6rD2Meo=","subType":"0"}},"keyId":7205967400469004293}},"$db":"admin"},"numYields":0,"reslen":681,"locks":{},"protocol":"op_msg","durationMillis":0}}
{"t":{"$date":"2023-03-17T10:47:21.486+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn14256","msg":"Slow query","attr":{"type":"command","ns":"admin.$cmd","command":{"hello":1,"helloOk":true,"topologyVersion":{"processId":{"$oid":"641410c417541421e074328b"},"counter":4},"maxAwaitTimeMS":10000,"$db":"admin","$readPreference":{"mode":"primaryPreferred"}},"numYields":0,"reslen":788,"locks":{},"protocol":"op_msg","durationMillis":0}}
{"t":{"$date":"2023-03-17T10:47:21.486+00:00"},"s":"D1", "c":"REPL",     "id":21342,   "ctx":"conn14256","msg":"Waiting for an isMaster response from a topology change or until deadline","attr":{"deadline":{"$date":"2023-03-17T10:47:31.486Z"},"currentTopologyVersionCounter":4}}
{"t":{"$date":"2023-03-17T10:47:21.670+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn16","msg":"Slow query","attr":{"type":"command","ns":"admin.$cmd","command":{"isMaster":1,"maxAwaitTimeMS":10000,"topologyVersion":{"processId":{"$oid":"641410c417541421e074328b"},"counter":4},"internalClient":{"minWireVersion":9,"maxWireVersion":9},"$db":"admin"},"numYields":0,"reslen":825,"locks":{},"protocol":"op_msg","durationMillis":0}}
{"t":{"$date":"2023-03-17T10:47:21.670+00:00"},"s":"D1", "c":"REPL",     "id":21342,   "ctx":"conn16","msg":"Waiting for an isMaster response from a topology change or until deadline","attr":{"deadline":{"$date":"2023-03-17T10:47:31.670Z"},"currentTopologyVersionCounter":4}}
{"t":{"$date":"2023-03-17T10:47:26.666+00:00"},"s":"I",  "c":"REPL",     "id":51801,   "ctx":"ReplWriterWorker-82","msg":"Applied op","attr":{"CRUD":{"lsid":{"id":{"$uuid":"e0251408-7c02-4109-89ca-a277c821a391"},"uid":{"$binary":{"base64":"pVfvZEWeAWCvrzs7QbVcVW2pM/2h4LPI35Lhs121Hgs=","subType":"0"}}},"txnNumber":26,"op":"u","ns":"canadaloctest77_bk_db.cron_status","ui":{"$uuid":"23980ddf-948a-4453-b88f-54a1dabf190c"},"o":{"$v":1,"$set":{"end_time":"Fri, 17 Mar 2023 10:47:19 UTC","status":1}},"o2":{"_id":{"$oid":"64144526a73332666c8d5580"}},"ts":{"$timestamp":{"t":1679050043,"i":5}},"t":99,"v":2,"wall":{"$date":"2023-03-17T10:47:23.873Z"},"stmtId":0,"prevOpTime":{"ts":{"$timestamp":{"t":0,"i":0}},"t":-1}},"durationMillis":58}}
{"t":{"$date":"2023-03-17T10:47:26.667+00:00"},"s":"I",  "c":"REPL",     "id":51801,   "ctx":"ReplWriterWorker-90","msg":"Applied op","attr":{"CRUD":{"lsid":{"id":{"$uuid":"61f1cac2-a81f-4a18-b1c3-58249a3dd79a"},"uid":{"$binary":{"base64":"pVfvZEWeAWCvrzs7QbVcVW2pM/2h4LPI35Lhs121Hgs=","subType":"0"}}},"txnNumber":21,"op":"u","ns":"moktest521_bk_db.cron_status","ui":{"$uuid":"1e205a38-b421-4924-bbfe-e948f8dacfba"},"o":{"$v":1,"$set":{"end_time":"Fri, 17 Mar 2023 10:47:19 UTC","status":1}},"o2":{"_id":{"$oid":"64144522a73332666c8d556c"}},"ts":{"$timestamp":{"t":1679050043,"i":6}},"t":99,"v":2,"wall":{"$date":"2023-03-17T10:47:23.921Z"},"stmtId":0,"prevOpTime":{"ts":{"$timestamp":{"t":0,"i":0}},"t":-1}},"durationMillis":59}}
{"t":{"$date":"2023-03-17T10:47:26.680+00:00"},"s":"I",  "c":"REPL",     "id":51801,   "ctx":"ReplWriterWorker-109","msg":"Applied op","attr":{"CRUD":{"lsid":{"id":{"$uuid":"07bf60ab-9fe9-42f8-807e-92fd0c044736"},"uid":{"$binary":{"base64":"pVfvZEWeAWCvrzs7QbVcVW2pM/2h4LPI35Lhs121Hgs=","subType":"0"}}},"txnNumber":11,"op":"i","ns":"moktest1059_bk_db.customer_reviews","ui":{"$uuid":"948ac792-5868-4c91-9021-2574952356a3"},"o":{"_id":{"$oid":"64144537a73332666c8d560f"},"booking_id":796,"dont_send_notification":false,"auth_token":"7HFcpuFxLmGnXk75kRNd"},"ts":{"$timestamp":{"t":1679050043,"i":2}},"t":99,"v":2,"wall":{"$date":"2023-03-17T10:47:26.607Z"},"stmtId":0,"prevOpTime":{"ts":{"$timestamp":{"t":0,"i":0}},"t":-1}},"durationMillis":68}}
{"t":{"$date":"2023-03-17T10:47:26.736+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn14277","msg":"Slow query","attr":{"type":"command","ns":"moktest281_bk_db.master_templates","command":{"find":"master_templates","filter":{"_id":{"$in":[{"$oid":"63ecb3ff2a1684f94b147d08"},{"$oid":"63ecb3ff2a1684f94b147d08"},{"$oid":"63ecb3ff2a1684f94b147d08"},{"$oid":"63ecb3ff2a1684f94b147d08"},{"$oid":"63ecb3ff2a1684f94b147d08"},{"$oid":"63ecb3ff2a1684f94b147d08"}]},"status":true},"batchSize":1,"singleBatch":true,"maxTimeMS":1000,"$readPreference":{"mode":"secondaryPreferred"},"readConcern":{"level":"local"},"$db":"moktest281_bk_db"},"planSummary":"IXSCAN { _id: 1 }","keysExamined":1,"docsExamined":1,"cursorExhausted":true,"numYields":0,"nreturned":1,"queryHash":"4CBEC556","planCacheKey":"9B7BDF92","locks":{"FeatureCompatibilityVersion":{"acquireCount":{"r":1}},"ReplicationStateTransition":{"acquireCount":{"w":1}},"Global":{"acquireCount":{"r":1}},"Database":{"acquireCount":{"r":1}},"Collection":{"acquireCount":{"r":1}},"Mutex":{"acquireCount":{"r":1}}},"readConcern":{"level":"local","provenance":"clientSupplied"},"storage":{"data":{"bytesRead":11460,"timeReadingMicros":10}},"protocol":"op_msg","durationMillis":6}}
{"t":{"$date":"2023-03-17T10:47:26.787+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn14277","msg":"Slow query","attr":{"type":"command","ns":"moktest287_bk_db.master_templates","command":{"find":"master_templates","filter":{"_id":{"$in":[{"$oid":"63ecb3ff2a1684f94b147d08"},{"$oid":"63ecb3ff2a1684f94b147d08"},{"$oid":"63ecb3ff2a1684f94b147d08"},{"$oid":"63ecb3ff2a1684f94b147d08"},{"$oid":"63ecb3ff2a1684f94b147d08"},{"$oid":"63ecb3ff2a1684f94b147d08"}]},"status":true},"batchSize":1,"singleBatch":true,"maxTimeMS":1000,"$readPreference":{"mode":"secondaryPreferred"},"readConcern":{"level":"local"},"$db":"moktest287_bk_db"},"planSummary":"IXSCAN { _id: 1 }","keysExamined":1,"docsExamined":1,"cursorExhausted":true,"numYields":0,"nreturned":1,"queryHash":"4CBEC556","planCacheKey":"9B7BDF92","locks":{"FeatureCompatibilityVersion":{"acquireCount":{"r":1}},"ReplicationStateTransition":{"acquireCount":{"w":1}},"Global":{"acquireCount":{"r":1}},"Database":{"acquireCount":{"r":1}},"Collection":{"acquireCount":{"r":1}},"Mutex":{"acquireCount":{"r":1}}},"readConcern":{"level":"local","provenance":"clientSupplied"},"storage":{"data":{"bytesRead":11460,"timeReadingMicros":10}},"protocol":"op_msg","durationMillis":6}}```
sahil garg
  • 89
  • 5
  • Which MongoDB version do you use? Do you use x.509 certificates for replica set membership authentication? – Wernfried Domscheit Mar 17 '23 at 07:26
  • I am using mongoDb version 4.4.1, I have setup replica with key files https://www.mongodb.com/docs/manual/tutorial/deploy-replica-set-with-keyfile-access-control/#:~:text=Start%20each%20member%20of%20the,line%20option%20or%20the%20security. – sahil garg Mar 17 '23 at 08:19
  • There should be much more logs when you say "the MongoDB stops", i.e. it shut down. – Wernfried Domscheit Mar 17 '23 at 09:01
  • MongoDb doesn't shut down, but yes there was a lot of logs running, because I am working on a large set of data. In which a lot of db query are running in parallel. Do you have any idea how can i avoid these situations? – sahil garg Mar 17 '23 at 09:08
  • 1
    I doubt the provided logs says something wrong. If I recall correctly, it's simple logs line related to inner driver machinery, you should provide more logs or details – dododo Mar 17 '23 at 09:58
  • With [jq](https://www.mongodb.com/docs/manual/reference/log-messages/#log-parsing-examples) you can filter and format the log messages. And you may set [logging level](https://www.mongodb.com/docs/manual/reference/configuration-options/#mongodb-setting-systemLog.verbosity) to default. – Wernfried Domscheit Mar 17 '23 at 10:06
  • Hello @dododo, sorry for the late reply, I have added some more logs in the post that i am facing continously. – sahil garg Mar 20 '23 at 04:02
  • I don't see any delays in these logs. Also, I don't see any suspicious lines too. Logic about `JournalFlusher` is probably internal server logic, I doubt it's anything important, but in any case it doesn't give any delays based on timings – dododo Mar 20 '23 at 11:20

0 Answers0