7

I have a couple of servers that are approaching capacity, however I can't seem to find where the space is being consumed? When I run:

[root@server /]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
                       40G   38G   76M 100% /
tmpfs                 929M     0  929M   0% /dev/shm
/dev/sda1             485M   32M  428M   7% /boot

I can see that the / mount is filling up, so I then run:

[root@server /]# sudo ls | xargs du -hs
5.8M    bin
22M     boot
192K    dev
25M     etc
36K     home
122M    lib
19M     lib64
16K     lost+found
4.0K    media
4.0K    mnt
4.0K    opt
0       proc
124K    root
14M     sbin
0       selinux
4.0K    srv
0       sys
54M     tmp
749M    usr
61M     var

and can't see any real directories that are using heaps of space. The biggest one is /usr at 749M.

Can someone please suggest how I can find out where all this space is being chewed up? Thanks.

cpjones44
  • 160
  • 2
  • 8

1 Answers1

10

Often times when something like this happens, it is due to some process keeping filehandles open on deleted (unlinked) files. If this is the case, you can find them by using lsof, for example:

$ sudo lsof +L1
COMMAND PID USER   FD   TYPE DEVICE SIZE/OFF NLINK    NODE NAME
tuned   869 root    7u   REG  202,1     4096     0 4194380 /tmp/ffiGemRJt (deleted)
$

If you find deleted files are consuming much of your space, then it will probably be necessary to restart the offending process that is keeping the filehandles open.

guzzijason
  • 1,410
  • 8
  • 18
  • 2
    Thanks. That was it. ngninx on the servers was chewing up the space. Restarted the services and it's all good. – cpjones44 Oct 09 '18 at 01:38
  • 1
    This is classic. It happens to folks all of the time. https://serverfault.com/questions/881501/lsof-shows-tmp-growing-file-marked-as-deleted is one of many examples. – chicks Oct 09 '18 at 18:36