7
# df -h /
Filesystem            Size  Used Avail Use% Mounted on
rootfs                9.9G  7.2G  2.2G  77% /

# du -hx --max-depth=0 /
3.2G    /

As you can see, df says 7.2GB is used, but du can only find 3.2GB of it. The server has been rebooted since I noticed this, so it's not a deleted file. Additionally, lsof doesn't show me anything interesting. What else could it be?

womble
  • 96,255
  • 29
  • 175
  • 230
wes
  • 91
  • 1
  • 2
  • 4

4 Answers4

13

I was having the exact same issue on my ext4 system and just wanted to post my solution for future reference. When my drive initially filled out, I deleted a bunch of logs out of /var/log. That cleared up a couple GB, but within a few days I ran out of space again, and du -h and "mount --bind / /mnt" did not point to the culprit. What finally got it was when I ran lsof.

lsof
...
rsyslogd   1766      root    2w      REG                9,1   12672375940     264014 /var/log/messages (deleted)
...

When I had deleted the messages log file, the rsyslog service still kept it open, but hidden. Running a "touch /var/log/messages;service rsyslog restart" cleared up the problem and my disk space was reclaimed.

The lsof output can be a bit overwhelming, especially if you have a busy system (it was over 1000 lines long on mine). If you grep for "deleted" in the lsof output, it should help to pinpoint the problem process.

John P
  • 1,679
  • 6
  • 38
  • 59
  • Same here, except in my case it was both `/var/log/messages` and `var/log/secure` (both held by `rsyslogd`). Thanks! – dgould Aug 12 '19 at 18:43
6

Since you're using the -x option, I assume you have other filesystems mounted? It could be that you have another partition mounted on top of a directory that wasn't empty.

DerfK
  • 19,493
  • 2
  • 38
  • 54
  • You save me a lot of time, I have a backup script which performs its operations by anacron from ssd to hdd, and when I temporary removed hdd it had pushed a backup to hdd mount point. And of course I didn't catch this. – xwild Jul 28 '17 at 14:52
5

Make sure all folders "behind" your mountpoints are empty. Out of experience i'd say most probably you hide some data behind a mountpoint.

you can check what is in the folder behind the mountpoint without having to unmount the disk/partitions (which can be nice if for example you would have to unmount /usr). Just do a

# mount --bind / /mnt
# du -shx /mnt

also as a side note:

# du -shx /

does the same as your du... but is quite a bit shorter (-s stands for summarize)

Marcel G
  • 2,269
  • 15
  • 24
  • this was exactly it, thanks. the --bind flag was also very helpful - I would have had a much harder time finding it without it. – wes Aug 11 '11 at 15:31
4

There could be a couple of problems: (1) a process has a large file or number of files open that have been deleted, or (2) you might have some type of filesystem problem remedied by an fsck of the drive. However, this can only be done when the drive is not mounted. Several Linux provide a method of FULL fsck of the root file at boot time by:

 touch /forcefsck

and then rebooting. I know this exists in SuSE and RedHat varients.

mdpc
  • 11,856
  • 28
  • 53
  • 67