Ubuntu root drive running out of space, I can't find source through du or lsof

Question

The root drive on an Ubuntu 15.10 machine is nearly out of space, but I can't find the source. The drive that's running out of space is sdb2, 313M of 51G available. The filesystem is ext4.

Here is the sudo du -h / --max-depth=1 output:

Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           789M  9.4M  780M   2% /run
/dev/sdb2        51G   48G  313M 100% /
tmpfs           3.9G   12K  3.9G   1% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sdb1       511M  3.4M  508M   1% /boot/efi
tmpfs           789M  8.0K  789M   1% /run/user/1000
/dev/sda1       239G  122M  239G   1% /media/DATA

But I can't find any large files. Total usage at / seems to be only 3.4 G. Here is the output from sudo du -h / --max-depth=1:

4.0K    /mnt
188K    /tmp
406M    /home
339M    /var
8.1M    /etc
361M    /lib
du: cannot access ‘/proc/7626/task/7626/fd/4’: No such file or directory
du: cannot access ‘/proc/7626/task/7626/fdinfo/4’: No such file or directory
du: cannot access ‘/proc/7626/fd/3’: No such file or directory
du: cannot access ‘/proc/7626/fdinfo/3’: No such file or directory
0    /proc
13M    /bin
du: cannot access ‘/run/user/1000/gvfs’: Permission denied
9.4M    /run
1.6M    /root
4.0K    /lib64
16K    /lost+found
0    /sys
1.1M    /media
12K    /dev
222M    /opt
2.0G    /usr
62M    /boot
9.5M    /sbin
4.0K    /cdrom
8.0K    /srv
3.4G    /

I found a similar question here: Out of disk space, what's the source?

In that case, it looks like the problem was caused by a deleted log that somehow was not closed by a running process, and the way to find it was to run sudo lsof | grep deleted. In my case, the output is

lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.

Also, the problem persists after rebooting the system, so it's unlikely that that is the cause.

Another suggested solution is to unmount /var/lib/ureadahead/debugfs, but I'm hesitant to do that.

What else could be wrong?

Check the filesystem for errors, and update to a supported Ubuntu release. — Michael Hampton, Jan 11 '17 at 05:15
Is there a way to check the filesystem over TeamViewer, which I'm guessing requires to keep it mounted? — biggvsdiccvs, Jan 11 '17 at 05:39
What filesystem is used for root partition? Please show output of `mount` command. — Fedor Dikarev, Jan 11 '17 at 06:37
For reference you can use `lsof -e /run/user/1000/gvfs` to exclude that (or other filesystems). However if you have rebooted that's unlikely to help here . — user9517, Jan 11 '17 at 06:47
just run "du -sh *" when you are in the root folder. Do not use --max-depth flag — Orphans, Jan 11 '17 at 07:00
@Orphans go and try `du -sh` and `du --max-depth=1` you'll see why it's normally useful in this situation. — user9517, Jan 11 '17 at 07:03
@Hanginoninquietdesperation, well it seems that I have missunderstood what --max-depth does. But the output is basically the same, except that du -sh shows it in human — Orphans, Jan 11 '17 at 07:38
@Orphans looks like I didn't see the `*` nestled up against ypu `"` — user9517, Jan 11 '17 at 07:55
Why are you looking for large files? You might have a directory full of fairly small files, but added together, they amount to quite a bit. `du -x / | sort -nr >disk-use.txt` Then look at that file and your culprit ought to be near the top. — Monty Harder, Jan 11 '17 at 17:41

score 21 · Accepted Answer · answered Jan 11 '17 at 07:02

21

Well, it's only a guess but it may works: I think that user once forget to mount /dev/sda1 as /media/DATA and all data was written on /dev/sdb2 instead of /dev/sda1.

To check this, please, unmount /media/DATA and check files and folders under this path.

answered Jan 11 '17 at 07:02

Fedor Dikarev

736
4
10

Thanks, that's what it was! Except I think it might have been a bad drive connection rather than anyone forgetting to mount anything. That whole setup is unreliable and I'm not sure how that drive is mounted to `/media/DATA` -- it's not in `/etc/fstab`. – biggvsdiccvs Jan 11 '17 at 07:28
If you don't want to unmount the filesystem (for example, if you don't want to shut down the processes that are using it), you can use a [bind mount](http://unix.stackexchange.com/a/4428/20276) to see (and delete) the files hidden by the mount. – Johnny Jan 11 '17 at 21:34

score 8 · Answer 2 · edited Jan 11 '17 at 13:15

8

I regularly use 'ncdu' for that, it's small enough to still get it installed.

sudo apt-get install ncdu

Just make sure you run it as root or via sudo:

sudo ncdu /

edited Jan 11 '17 at 13:15

A.L

126
1
8

answered Jan 11 '17 at 09:18

Roy Jacobs

223
1
6

1

Wouldn't have helped in this case though. – Gerald Schneider Jan 11 '17 at 09:22

score 3 · Answer 3 · answered Jan 11 '17 at 19:14

When you want to know where space is being used in a particular filesystem, you can use this command to find the 20 largest directories, which can help you locate where the most space is used,

du -m / |sort -n |tail -20

But the root filesystem is more difficult, because all filesystems are mounted to root. But the -x (--one-file-system) argument will report only the desired filesystem,

du -m -x / |sort -n |tail -20

score 1 · Answer 4 · edited Jan 11 '17 at 17:25

1

df -h *.*

Might help.

Traverses directories and sums up bytes used.

edited Jan 11 '17 at 17:25

Community

1

answered Jan 11 '17 at 15:18

user394616

11
1

Ubuntu root drive running out of space, I can't find source through du or lsof

4 Answers4