1

I have a host that runs CentOS-6 with Linux 2.6.32-696.18.7.el6.x86_64 (yes, pretty old) and I ran out of disk space on the root filesystem.

The filesystem's size is 16G, du shows about 3G being used by files. Of course, I ran lsof +L1, found processes that held the big log-files that had been unlinked before (yes, there were a couple of such processes, so I killed them), but the filesystem is still full and anything can write to it (no, inodes are fine, there are no available blocks).

lsof +L1 still shows some files, but these files don't occupy too much size, so I'm pretty sure I can ignore that:

COMMAND     PID      USER   FD   TYPE DEVICE SIZE/OFF NLINK   NODE NAME
init          1      root    9r   REG  253,0 10406312     0 131692 /var/lib/sss/mc/initgroups (deleted)
pickup     2761   postfix    8r   REG  253,0 10406312     0 134123 /var/lib/sss/mc/initgroups (deleted)
dbus-daem  5232      dbus    4r   REG  253,0 10406312     0 131692 /var/lib/sss/mc/initgroups (deleted)
hald       5271 haldaemon   19r   REG  253,0 10406312     0 131692 /var/lib/sss/mc/initgroups (deleted)
ntpd       5528       ntp    4r   REG  253,0 10406312     0 131730 /var/lib/sss/mc/initgroups (deleted)
qmgr       5631   postfix    8r   REG  253,0 10406312     0 131730 /var/lib/sss/mc/initgroups (deleted)
crond      5655      root    8r   REG  253,0  8406312     0 131792 /var/lib/sss/mc/passwd (deleted)
sssd       9042      root   16r   REG  253,0 10406312     0 134123 /var/lib/sss/mc/initgroups (deleted)
sssd_be    9045      root   18r   REG  253,0 10406312     0 134123 /var/lib/sss/mc/initgroups (deleted)
qemu-kvm   9606      root  txt    REG  253,0  4056888     0 397695 /usr/libexec/qemu-kvm (deleted)
qemu-kvm  16799      root  txt    REG  253,0  4056888     0 397695 /usr/libexec/qemu-kvm (deleted)
qemu-kvm  17419      root  txt    REG  253,0  4056888     0 397695 /usr/libexec/qemu-kvm (deleted)
polkitd   28420      root    8r   REG  253,0 10406312     0 134123 /var/lib/sss/mc/initgroups (deleted)
qemu-kvm  28802      root  txt    REG  253,0  4056888     0 397695 /usr/libexec/qemu-kvm (deleted)
qemu-kvm  30171      root  txt    REG  253,0  4056888     0 397695 /usr/libexec/qemu-kvm (deleted)
qemu-kvm  32110      root  txt    REG  253,0  4056888     0 397695 /usr/libexec/qemu-kvm (deleted)
qemu-kvm  32419      root  txt    REG  253,0  4056888     0 397695 /usr/libexec/qemu-kvm (deleted)
qemu-kvm  32563      root  txt    REG  253,0  4056888     0 397695 /usr/libexec/qemu-kvm (deleted)
qemu-kvm  34762      root  txt    REG  253,0  4056888     0 397695 /usr/libexec/qemu-kvm (deleted)
qemu-kvm  37823      root  txt    REG  253,0  4056888     0 397695 /usr/libexec/qemu-kvm (deleted)
qemu-kvm  39088      root  txt    REG  253,0  4056888     0 397695 /usr/libexec/qemu-kvm (deleted)
qemu-kvm  41613      root  txt    REG  253,0  4056888     0 397695 /usr/libexec/qemu-kvm (deleted)
nrpe      45821      nrpe    4r   REG  253,0 10406312     0 134123 /var/lib/sss/mc/initgroups (deleted)
sshd      52715      root    4w   REG  253,0 10406312     0 134123 /var/lib/sss/mc/initgroups (deleted)
qemu-kvm  55329      root  txt    REG  253,0  4056888     0 397695 /usr/libexec/qemu-kvm (deleted)

What else should I try to do (I'm pretty sure that reboot will help, but I'd like to avoid doing that)?

Thanks!

Volodymyr Melnyk
  • 577
  • 1
  • 5
  • 19
  • 1
    Just making sure: you killed the processes, but did you delete the files as well? – Konrad Gajewski Jan 13 '20 at 21:11
  • I'm gonna go out on a limb and suggest that you may space issues relating to sparse files. That could account for a difference in reporting (what does 'df' show?). Sparse files would also be a likely thing to find a VM environment. 'ls -lsh' will be useful to show actual used space rather than apparent space. – Cameron Kerr Jan 13 '20 at 22:00
  • @KonradGajewski, yes, for sure, I unlinked them with `rm` before to kill the processes, so they were been shown by `lsof +L1` as "deleted". – Volodymyr Melnyk Jan 14 '20 at 07:16
  • @CameronKerr, thanks for pointing on that, but all sparse files are located on a separate XFS filesystem, there are no VM's images on the root filesystem, that's why I didn't consider that it could be related to sparse files. – Volodymyr Melnyk Jan 14 '20 at 07:18

0 Answers0