0

We are using MongoDB version 4.2.11. We have one primary and two secondary nodes of MongoDB on Kubernetes. During load testing phase, despite having ample disk space we got a WiredTiger exception saying disk space is full.

2022-05-01T19:57:03.058+0000 W  FTDC     [ftdc] Uncaught exception in 'FileStreamFailed: Failed to write to interim file buffer for full-time diagnostic data capture: /var/data/mon      godb/diagnostic.data/metrics.interim.temp' in full-time diagnostic data capture subsystem.Shutting down the full-time diagnostic data capture subsystem.
10622 2022-05-01T19:57:04.534+0000 I  NETWORK  [conn12883] received client metadata from 127.0.0 .1:33116 conn12883: { driver: { name: "nodejs", version: "2.2.36" }, os: { type: "Linux",name: "linux", architecture: "x64", version: "5.4.0-87-generic" }, platform: "Node.js v12.16.1, LE, mongodb-core: 2.1.20" }
10623 2022-05-01T19:57:04.607+0000 E  STORAGE  [WTCheckpointThread] WiredTiger error (28) [1651435024:607232][80:0x7f0b7fe7e700], file:WiredTiger.wt, WT_SESSION.checkpoint: __posix_file_      write, 539: /var/data/mongodb/WiredTiger.turtle.set: handle-write: pwrite: failed to write 1262 bytes at offset 0: No space left on device Raw: [1651435024:607232][80:0x7f0b7fe7e700], file:WiredTiger.wt, WT_SESSION.checkpoint: __posix_file_write, 539: /var/data/mongodb/      WiredTiger.turtle.set: handle-write: pwrite: failed to write 1262 bytes at offset 0: No space left on device


2022-05-01T19:57:04.608+0000 E  STORAGE  [WTCheckpointThread] WiredTiger error (28) [16514      35024:608377][80:0x7f0b7fe7e700], file:WiredTiger.wt, WT_SESSION.checkpoint: __posix_file_      write, 539: /var/data/mongodb/WiredTiger.turtle.set: handle-write: pwrite: failed to write       1262 bytes at offset 0: No space left on device Raw: [1651435024:608377][80:0x7f0b7fe7e70      0], file:WiredTiger.wt, WT_SESSION.checkpoint: __posix_file_write, 539: /var/data/mongodb/      WiredTiger.turtle.set: handle-write: pwrite: failed to write 1262 bytes at offset 0: No sp      ace left on device
10625 2022-05-01T19:57:04.608+0000 E  STORAGE  [WTCheckpointThread] WiredTiger error (28) [16514      35024:608480][80:0x7f0b7fe7e700], file:WiredTiger.wt, WT_SESSION.checkpoint: __wt_turtle_u      pdate, 391: WiredTiger.turtle: fatal turtle file update error: No space left on device Raw      : [1651435024:608480][80:0x7f0b7fe7e700], file:WiredTiger.wt, WT_SESSION.checkpoint: __wt_      turtle_update, 391: WiredTiger.turtle: fatal turtle file update error: No space left on de      vice

2022-05-01T19:57:04.608+0000 E  STORAGE  [WTCheckpointThread] WiredTiger error (-31804) [1      651435024:608496][80:0x7f0b7fe7e700], file:WiredTiger.wt, WT_SESSION.checkpoint: __wt_pani      c, 489: the process must exit and restart: WT_PANIC: WiredTiger library panic Raw: [1`enter code here`5024:608496][80:0x7f0b7fe7e700], file:WiredTiger.wt, WT_SESSION.checkpoint: __wt_panic, 489: the process must exit and restart: WT_PANIC: WiredTiger library panic
2022-05-01T19:57:04.608+0000 F [WTCheckpointThread] Fatal Assertion 50853 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 486
2022-05-01T19:57:04.608+0000 F[WTCheckpointThread] \n\n***aborting after fassert() failure\n\n
Wernfried Domscheit
  • 54,457
  • 9
  • 76
  • 110
Vineel
  • 1,630
  • 5
  • 26
  • 47
  • The error is quite clear: "No space left on device" Note, the oplog also takes considerable amount of space. How much space is left on your drive and how big is your data? – Wernfried Domscheit May 03 '22 at 06:37
  • @WernfriedDomscheit I did df -h and verified that there is still space left. You mean that oplog (set to 30GB) also takes space on the disk? But when I logged into the system, I did see that there is still ample space. – Vineel May 03 '22 at 06:39
  • Yes, oplog also takes space. Note, the oplog can grow past its configured size limit to avoid deleting the majority commit point. – Wernfried Domscheit May 03 '22 at 06:55

0 Answers0