How can I reclaim bytes from a nearly full, 850G mongodb partition?

Question

Woke up this morning to find out MongoDB grew to all available disk space for some reason. I cannot repair the db to compact it, as I have no space left. I can't even start postgres to remove an old database which would free up a few gigs more. I literally have zero free disk space.

I did delete a 9GB file but the space is not showing up at all when I "df -h".

I have to clear up space on the drive asap as its a production server and we're down. What to do?

score 7 · Answer 1 · answered Mar 21 '11 at 11:12

7

the space will not show up until all processes release the file. Try using lsof to see what's using the file.

answered Mar 21 '11 at 11:12

Sirex

5,499
2
33
54

The file i deleted was an old db sql dump. It was not used by anything. – Harel Mar 21 '11 at 11:44
To make things worst, i droped the bad mongodb collection but i cannot run repairDatabase which would reclaim the space, because I don't have any space left! Its a catch22 – Harel Mar 21 '11 at 11:45
Did you double check lsof to see if there are any open file handles? – Bart Silverstrim Mar 21 '11 at 12:22
Yes. I also stopped all the services I could stop. No matter what I deleted, df -h shows 0 bytes left on drive (even after i run 'sync' or reboot). – Harel Mar 21 '11 at 12:24

score 3 · Accepted Answer · edited Aug 05 '14 at 17:16

Although the answer provided is valid in many cases, in my scenario I ended up falling into the ext3 5% reserved root space. I could have recovered this space had I known about it at the time. I ended up using a USB drive as a repair path to mongodb reclaiming back 850GB.

This is how to recover that ext3 5% should anyone be interested:

http://www.andremiller.net/content/recovering-reserved-space-ext2-and-ext3-filesystems

How can I reclaim bytes from a nearly full, 850G mongodb partition?

2 Answers2