0

Summary

I'm running a debian on a dedicated server hosting multiples things.

The root partition of the server is +/-20GB wide and 95% used. This usage is growing over time (from 90% to 95% in 2 weeks). The tools I found to track space usage only show a total of +/- 10GB used on the root partition.

du / dh results

$ df
Filesystem      1K-blocks      Used Available Use% Mounted on
udev              8202864         0   8202864   0% /dev
tmpfs             1641464    173948   1467516  11% /run
/dev/sda3        20026236  17949976   1035928  95% /
tmpfs             8207312         4   8207308   1% /dev/shm
tmpfs                5120         0      5120   0% /run/lock
tmpfs             8207312         0   8207312   0% /sys/fs/cgroup
/dev/sda2          498532     36207    432066   8% /boot
/dev/sda4      1901536420 906713452 898207496  51% /home
tmpfs             1641460         0   1641460   0% /run/user/1004

875G    total
875G    .
865G    ./home
8.9G    ./var
923M    ./usr
170M    ./run
47M     ./lib
34M     ./boot
22M     ./tmp
21M     ./sbin
12M     ./bin
8.6M    ./etc
100K    ./root
56K     ./opt
12K     ./srv
8.0K    ./media
4.0K    ./mnt
4.0K    ./lib64
4.0K    ./dev
0       ./sys
0       ./proc

As you can see, total used is 875GB, minus the 865GB of /home it leave 10Go used. The df however indicate that the usage is about 20GB :/ All listed folders in the second snippet DID NOT change size during those 15 days.

I have no idea what's going on, any help is welcome ! Thanks for your time :)

PS : We just rebooted the server in desperation and it felt back to 56% usage it should normally have ... I'd still like to know what the fuck is going on ! The server had about 1 year uptime at the restart, no idea if that helps

Slivo
  • 1
  • 1

1 Answers1

0

We just rebooted the server in desperation and it felt back to 56% usage it should normally have ... I'd still like to know what the fuck is going on

You deleted a file that some application still had open. Using rm on a file simply removes the link to the file. But a file is not actually removed until no applications have it open.

One of the common causes is that some application didn't roll/rotate the logs correctly, and has an open handle to the 'deleted' version of the log instead of what should be the current log.

Anyway, what you have to do is just do some looking around with lsof or other similar tools to figure out which application had open, but removed files.

Zoredache
  • 130,897
  • 41
  • 276
  • 420
  • I tried some lsof commands before restart and they were returning nothing at all. I will try it again when space will start fill up again ! Thanks for your help – Slivo Apr 28 '20 at 09:40
  • The server started filling up again, lsof found deleted files from mysql. Restarting mysql free the space ! – Slivo Jul 03 '20 at 09:43
  • That seems odd. Never seen mysql cause that kind of error. But then I have mostly switched over to mariadb. – Zoredache Jul 03 '20 at 22:10
  • We're on maria DB too : `mysql Ver 15.1 Distrib 10.4.7-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2`. Here is the kind of file I found with lsof : https://pastebin.com/35jb09NX – Slivo Jul 05 '20 at 12:44