4

I upgraded an older server from 13.10 -> 14.04 -> 14.10 -> 15.10 yesterday. Today I'm seeing that 100% of the disk space is being used but I can't track down where the usage is.

When I run du -h --max-depth=1 / I get the following log which seems to indicate the root path is full, but I can't tell what the source is. 

du: cannot read directory ‘/proc/4390/task/4390/ns’: Permission denied
du: cannot read directory ‘/proc/4390/fd’: Permission denied
du: cannot read directory ‘/proc/4390/map_files’: Permission denied
du: cannot read directory ‘/proc/4390/fdinfo’: Permission denied
du: cannot read directory ‘/proc/4390/ns’: Permission denied
du: cannot read directory ‘/proc/4392/task/4392/fd’: Permission denied
du: cannot read directory ‘/proc/4392/task/4392/fdinfo’: Permission denied
du: cannot read directory ‘/proc/4392/task/4392/ns’: Permission denied
du: cannot read directory ‘/proc/4392/fd’: Permission denied
du: cannot read directory ‘/proc/4392/map_files’: Permission denied
du: cannot read directory ‘/proc/4392/fdinfo’: Permission denied
du: cannot read directory ‘/proc/4392/ns’: Permission denied
du: cannot access ‘/proc/4393/task/4393/fd/3’: No such file or directory
du: cannot access ‘/proc/4393/task/4393/fdinfo/3’: No such file or directory
du: cannot access ‘/proc/4393/fd/4’: No such file or directory
du: cannot access ‘/proc/4393/fdinfo/4’: No such file or directory
0       /proc
du: cannot read directory ‘/sys/kernel/debug’: Permission denied
0       /sys
4.0K    /mnt
1.5G    /usr
du: cannot read directory ‘/root’: Permission denied
4.0K    /root
4.0K    /srv
104M    /home
7.6M    /sbin
12M     /bin
47M     /boot
du: cannot read directory ‘/var/lib/snmp/mib_indexes’: Permission denied
du: cannot read directory ‘/var/lib/php5’: Permission denied
du: cannot read directory ‘/var/lib/mysql’: Permission denied
du: cannot read directory ‘/var/lib/sudo’: Permission denied
du: cannot read directory ‘/var/lib/nginx/proxy’: Permission denied
du: cannot read directory ‘/var/lib/nginx/uwsgi’: Permission denied
du: cannot read directory ‘/var/lib/nginx/scgi’: Permission denied
du: cannot read directory ‘/var/lib/nginx/body’: Permission denied
du: cannot read directory ‘/var/lib/nginx/fastcgi’: Permission denied
du: cannot read directory ‘/var/log/unattended-upgrades’: Permission denied
du: cannot read directory ‘/var/log/mysql’: Permission denied
du: cannot read directory ‘/var/log/nginx’: Permission denied
du: cannot read directory ‘/var/spool/rsyslog’: Permission denied
du: cannot read directory ‘/var/spool/cron/atspool’: Permission denied
du: cannot read directory ‘/var/spool/cron/crontabs’: Permission denied
du: cannot read directory ‘/var/spool/cron/atjobs’: Permission denied
du: cannot read directory ‘/var/cache/ldconfig’: Permission denied
du: cannot read directory ‘/var/cache/nginx’: Permission denied
18G     /var
8.0K    /media
4.0K    /lib64
du: cannot read directory ‘/run/log/journal/41cc217bd9afc046de32243850804999’: Permission denied
du: cannot read directory ‘/run/systemd/inaccessible’: Permission denied
1.5M    /run
359M    /lib
0       /dev
4.0K    /opt
20G     /

When I ls -la in / I see this:

total 100
drwxr-xr-x  22 root root  4096 Jul 16 00:21 .
drwxr-xr-x  22 root root  4096 Jul 16 00:21 ..
drwxr-xr-x   2 root root  4096 Jul 16 00:16 bin
drwxr-xr-x   3 root root  4096 Jul 16 00:21 boot
drwxr-xr-x  13 root root 13680 Jul 16 18:46 dev
drwxr-xr-x 108 root root 12288 Jul 16 00:23 etc
drwxr-xr-x   3 root root  4096 Jul 15 23:23 home
lrwxrwxrwx   1 root root    32 Jul 16 00:18 initrd.img -> boot/initrd.img-4.2.0-42-generic
drwxr-xr-x  21 root root  4096 Jul 16 00:16 lib
drwxr-xr-x   2 root root  4096 Jul 16 00:14 lib64
drwx------   2 root root 16384 Oct 18  2012 lost+found
drwxr-xr-x   3 root root  4096 Jul 15 23:21 media
drwxr-xr-x   2 root root  4096 Oct  9  2012 mnt
drwxr-xr-x   2 root root  4096 Oct 17  2012 opt
dr-xr-xr-x 169 root root     0 Jul 16 18:46 proc
drwx------   5 root root  4096 Jul 15 23:21 root
drwxr-xr-x  19 root root   740 Jul 16 18:49 run
drwxr-xr-x   2 root root 12288 Jul 16 00:16 sbin
drwxr-xr-x   2 root root  4096 Oct 17  2012 srv
dr-xr-xr-x  13 root root     0 Jul 16 18:57 sys
drwxrwxrwt   7 root root  4096 Jul 16 19:04 tmp
drwxr-xr-x  10 root root  4096 Jul 15 23:23 usr
drwxr-xr-x  14 root root  4096 Jul 16 00:17 var
lrwxrwxrwx   1 root root    29 Jul 16 00:18 vmlinuz -> boot/vmlinuz-4.2.0-42-generic

Again, not seeing what's using the disk space.

What are some steps I should be doing here?

user
  • 4,335
  • 4
  • 34
  • 71
Geuis
  • 637
  • 3
  • 8
  • 20

2 Answers2

10

Run the following command to find free disk space information:

df -h

Then run the following command to find disk utilization information:

sudo du -shc /

(note the sudo, this should resolve the permission issues you were seeing)

When you find a directory taking up a considerable amount of disk space, cd to the directory and repeat step #2. Keep repeating this step until you identify the source of the disk space issue.

Paul Calabro
  • 530
  • 5
  • 12
  • 3
    Also, your ```/var``` directory is a 18 gigabytes. That might be worth looking at as well. – Paul Calabro Jul 16 '16 at 19:17
  • 6
    Yep, using sudo helped to clear out the noise. Found the cause. Had an old logging process that would write to a mongodb db maybe a dozen times a week. The process freaked out about something during/after the upgrade and wrote over 17gb of logs overnight. I'm not really using the old logger anyway so just removed the entire mongodb db. All smooth now! – Geuis Jul 16 '16 at 19:24
  • Glad to hear it worked out! – Paul Calabro Jul 16 '16 at 19:26
0

I would have commented on @Paul Calabro's answer if I had enough reputation on serverfault.

I also realise that the OP's happy that their problem is resolved but I thought it useful for those that are more nervous or at a novice level.

Would

sudo du -shc /*

not have been a better option as it would list each folder in the root directory?

However as @Paul then later refers to using cd then maybe

cd /

# position 1
sudo du -shc *

cd <into whatever folder you want drill down into>
# repeat from position 1

I was originally thinking about du -x but when this did not work i realised it was because du -x * is not stripping out the home folder because whilst it is indeed in a different filesystem the folder * expands to include home explicitly.

Further to make the output slightly easier to discern the large from the not so large replace du -h with du -BM, or for a coarser view du -BG, as then you would get a report something like

0M      /sys
0M      /mnt
1500M   /usr
0M      /root
0M      /srv
104M    /home
8M      /sbin
12M     /bin
47M     /boot
18000M  /var
0M      /media
0M      /lib64
2M      /run
359M    /lib

Now the /var folder noticeably sticks out as you don't have to pay attention to whether the units are K, M or G. I realise the original without -h or -BM would have also highlighted the size however it is possibly a case of too much detail or information overload. Are you really bothered if a folder uses 4 blocks or 400 blocks on a 20G device?

Whilst writing this I thought it was a pity there is not a -B% option which reports each line as a percentage of the total.

And finally for a bit of light relief...

Bragging rights

sudo du -scBP /

Extreme bragging rights

sudo du -scBE /

northern-bradley
  • 101
  • 1
  • 2