Reclaim free space from MongoDB 3.2

Question

Background: MongoDB ReplicaSet with 1 member and 1 arbiter is set up. It's necessary to free up disk space, because data in MongoDB is no more needed. All documents were removed with db.collection.remove({}) command. After db.runCommand ( { compact: 'collection', force:true} ), db.collection.reIndex() and db.repairDatabase() were ran, db.stats() looked like:

> db.stats()
{
    "db" : "database",
    "collections" : 2,
    "objects" : 6,
    "avgObjSize" : 167.83333333333334,
    "dataSize" : 1007,
    "storageSize" : 153042944,
    "numExtents" : 0,
    "indexes" : 5,
    "indexSize" : 45056,
    "ok" : 1
}

As you can see, there are just 6 documents in DB (in the second collections) with total storage size of 153MB. But, if we look at the disk space usage:

# du -h --max-depth=1 | grep mongodb
9.8G    ./mongodb

And inside mongodb directory:

ls -l
total 9365696
-rw-r----- 1 username usergroup      16384 Aug 19 14:19 collection-0-2472884588219438804.wt
-rw-r----- 1 username usergroup      36864 Aug 19 14:19 collection-0-4266045208498277842.wt
-rw-r----- 1 username usergroup      12288 Aug 24 15:55 collection-0--7009505821458556818.wt
-rw-r----- 1 username usergroup      49152 Aug 19 14:20 collection-0--7439052959034576211.wt
-rw-r----- 1 username usergroup      36864 Aug 19 14:20 collection-0-8676027872146699793.wt
-rw-r----- 1 username usergroup      36864 Aug 19 14:19 collection-2-2472884588219438804.wt
-rw-r----- 1 username usergroup      32768 Aug 23 13:54 collection-2-2911328926458913167.wt
-rw-r----- 1 username usergroup 9589645312 Aug 24 15:55 collection-4-2472884588219438804.wt
-rw-r----- 1 username usergroup      36864 Aug 19 14:19 collection-7--7439052959034576211.wt
-rw-r----- 1 username usergroup      36864 Aug 19 14:19 collection-9--7439052959034576211.wt
drwxr-x--- 2 username usergroup       4096 Aug 24 16:03 diagnostic.data
-rw-r----- 1 username usergroup       4096 Aug 24 15:54 index-10-561247476684508201.wt
-rw-r----- 1 username usergroup      36864 Jul 20 15:09 index-10--7439052959034576211.wt
-rw-r----- 1 username usergroup      36864 Aug 19 14:19 index-11--7439052959034576211.wt
-rw-r----- 1 username usergroup      16384 Mar  4 09:20 index-1-2472884588219438804.wt
-rw-r----- 1 username usergroup      36864 Jul 20 15:09 index-1-4266045208498277842.wt
-rw-r----- 1 username usergroup      36864 Aug 19 14:20 index-1--7439052959034576211.wt
-rw-r----- 1 username usergroup      16384 Jul  5 13:25 index-1-8676027872146699793.wt
-rw-r----- 1 username usergroup      36864 Jul 20 15:09 index-2-4266045208498277842.wt
-rw-r----- 1 username usergroup      16384 Mar  4 09:20 index-3-2472884588219438804.wt
-rw-r----- 1 username usergroup      16384 Aug 23 14:00 index-6-561247476684508201.wt
-rw-r----- 1 username usergroup      16384 Aug 23 14:00 index-7-561247476684508201.wt
-rw-r----- 1 username usergroup       4096 Aug 24 15:54 index-8-561247476684508201.wt
-rw-r----- 1 username usergroup      16384 Aug 19 14:19 index-8--7439052959034576211.wt
-rw-r----- 1 username usergroup       4096 Aug 24 15:54 index-9-561247476684508201.wt
drwxr-x--- 2 username usergroup       4096 Aug 24 15:55 journal
-rw-r----- 1 username usergroup      36864 Aug 24 15:55 _mdb_catalog.wt
-rwxr-x--- 1 username usergroup          6 Aug 19 14:19 mongod.lock
-rw-r----- 1 username usergroup      36864 Aug 24 15:56 sizeStorer.wt
-rw-r----- 1 username usergroup         95 Dec 30  2015 storage.bson
-rw-r----- 1 username usergroup         46 Dec 30  2015 WiredTiger
-rw-r----- 1 username usergroup        534 Dec 30  2015 WiredTiger.basecfg
-rw-r----- 1 username usergroup       4096 Aug 19 14:19 WiredTigerLAS.wt
-rw-r----- 1 username usergroup         21 Dec 30  2015 WiredTiger.lock
-rw-r----- 1 username usergroup        903 Aug 24 15:57 WiredTiger.turtle
-rw-r----- 1 username usergroup      94208 Aug 24 15:57 WiredTiger.wt

This string seems strange to me:

    -rw-r----- 1 username usergroup 9589645312 Aug 24 15:55 collection-4-2472884588219438804.wt

Question: What can be in mystery collection-4-2472884588219438804.wt file? Why is it not affected by compact, repairDatabase commands? Is there any way to force MongoDB empty this file or somehow reclaim it's space?

Update: With help of @james-wahlin we've figured out, that 9.8Gb is the Replica Set Oplog's size. But how can I force MongoDB to free up space, despite potential data loss for other Replica Set Members?

My guess is this is the local.oplog.rs collection (https://docs.mongodb.com/manual/core/replica-set-oplog/). Look at db stats in the 'local' database to confirm. — James Wahlin, Aug 24 '16 at 13:20
Yes, you was absolutely right! It's oplog.rs's file: `> db.oplog.rs.stats() { "ns" : "local.oplog.rs", "count" : 30036627, "size" : 7106989772, "avgObjSize" : 236, "storageSize" : 9589645312, "capped" : true, "max" : -1, "maxSize" : NumberLong("7050707456"), "sleepCount" : 0, "sleepMS" : 0, "wiredTiger" : { ..., "uri" : "statistics:table:collection-4-2472884588219438804",` — cleversokol, Aug 25 '16 at 13:08
@JamesWahlin, can I somehow force MongoDB to clean up this collections or empty it? I know, that there are no members to be synchronized and this data is no more needed. I always have an option to start sync with empty DB on the second member node, if I want. — cleversokol, Aug 25 '16 at 13:32
You can't "empty" the oplog as it is a fixed size, but you can resize it. See https://docs.mongodb.com/manual/tutorial/change-oplog-size/ — James Wahlin, Aug 25 '16 at 18:04
Thank you @JamesWahlin, I see that. But it's a pity. If I had only one day with intensive IO operations, which increased oplog's size, it'll stay big, despite the fact there will never(or very seldom) be such IO activity again. — cleversokol, Aug 26 '16 at 07:28
Not true. The oplog collection size is fixed at creation time. If you resize it, it will not grow unless you manually resize again. — James Wahlin, Aug 26 '16 at 13:49

score 0 · Answer 1 · answered Aug 24 '16 at 13:55

Based on the fact you removed all documents and the data is no longer needed:

Stop the mongod process

Remove all files from the ./mongodb directory manually

> use admin
switched to db admin
> db.shutdownServer()
2016-08-24T08:50:29.085-0500 I NETWORK  DBClientCursor::init call() failed server should be down...
2016-08-24T08:50:29.096-0500 I NETWORK  trying reconnect to 127.0.0.1:27017 (127.0.0.1) failed
2016-08-24T08:50:30.103-0500 W NETWORK  Failed to connect to 127.0.0.1:27017, reason: errno:10061 No connection could be made because the target machine actively refused it.
2016-08-24T08:50:30.104-0500 I NETWORK  reconnect 127.0.0.1:27017 (127.0.0.1) failed failed couldn't connect to server 127.0.0.1:27017 (127.0.0.1), connection attempt failed
>

# cd ./mongodb
# rm -rf *

When you restart mongo it will initialize the mongodb directory as needed.

Reclaim free space from MongoDB 3.2

1 Answers1