5

Output of df is:

[root@backup log]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGro  1889811408 1861658948         0 100% /
/dev/sda1               101086     16235     79632  17% /boot
tmpfs                  1815760         0   1815760   0% /dev/shm

So available block should be 28.152.460, but it's 0. I've been deleting a crap load of files, and used blocks is going down, but available remains at 0.

Output of df -i is: 

[root@backup log]# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/VolGro 487751680 238360803 249390877   49% /
/dev/sda1              26104      37   26067    1% /boot
tmpfs                 219784       1  219783    1% /dev/shm

So it's not a lack of inodes.

Output of lsof +L1 is:

[root@backup log]# /usr/sbin/lsof +L1
COMMAND     PID  USER   FD   TYPE DEVICE SIZE NLINK      NODE NAME
mysqld     2444 mysql    4u   REG  253,0    0     0 268795908 /tmp/ibSlaKC7 (deleted)
mysqld     2444 mysql    5u   REG  253,0    0     0 268795909 /tmp/ibhFuyGr (deleted)
mysqld     2444 mysql    6u   REG  253,0    0     0 268795910 /tmp/ibbNinKL (deleted)
mysqld     2444 mysql    7u   REG  253,0    0     0 268795911 /tmp/ibz1ia55 (deleted)
mysqld     2444 mysql   11u   REG  253,0    0     0 268795912 /tmp/ibM3IHvr (deleted)
crond      2549  root    3u   REG  253,0    5     0 248579098 /var/run/crond.pid (deleted)
yum-updat  2620  root   14w   REG  253,0    0     0 248611115 /var/run/yum.pid (deleted)
ssh       16256  root    0u   CHR  136,0          0         2 /dev/pts/0 (deleted)
ssh       16256  root    1u   CHR  136,0          0         2 /dev/pts/0 (deleted)
ssh       16256  root    2u   CHR  136,0          0         2 /dev/pts/0 (deleted)

I can't run 'du' because 99% of the disk usage is under /var/backups which contains probably ~100 million files (some idiot decided to rsync code from live server with subversion directories, so it's a lot of tiny files), so running 'du' would take days or weeks.

Does anyone have any suggestions on how to proceed?

jw.
  • 51
  • 1
  • 3

4 Answers4

10

If this is an ext filesystem, the default root reserved space would be 5% of 1889811408 blocks, or 94490570 blocks. In other words, you have about 66GB more to delete before df will report free space available.

Use tune2fs -m 1 /dev/mapper/VolGro to decrease the reserved amount to 1%, or -r NNNN to set it to a specific number of blocks. There needs to be enough reserved space that logging can continue even after users have "filled" the disk (though if you're filling the disk as root, this won't save you from problems when the drive is absolutely full)

Other filesystems probably have reserved blocks as well, but the commands to adjust these will differ.

DerfK
  • 19,493
  • 2
  • 38
  • 54
2

Run an fsck to see if the filesystem is damaged.

Sven
  • 98,649
  • 14
  • 180
  • 226
  • +1 - On linux this may also mean needing to unmount the affected filesystem (or touch the `.forcefsck` file in the FS root (IIRC) and then reboot). – voretaq7 Sep 08 '11 at 15:39
0

If you have deleted things like apache log files- it requires a apache restart before the space is "released"

AliGibbs
  • 2,323
  • 21
  • 34
  • If this were the case his blocks would not be showing up as free (`df` uses the kernel's idea of allocated space, which includes open file handles. `du` would incorrectly report that space as free since it just walks the directory tree (and only counts space from the inodes it finds)) – voretaq7 Sep 08 '11 at 15:36
  • lsof +L1 shows that there's only 5 bytes in deleted files – jw. Sep 09 '11 at 09:35
0

So available block should be 28.152.460, but it's 0. I've been deleting a crap load of files, and used blocks is going down, but available remains at 0.

I suspect that these files were still in use from your service when you delete it. So, simply restart the service, df command will update the available space.

quanta
  • 51,413
  • 19
  • 159
  • 217