Disk usage in LVM: Which directories contains most data?

Question

We have following volumes:

$ df -h
Filesystem                     Size  Used Avail Use% Mounted on
devtmpfs                       3.8G     0  3.8G   0% /dev
tmpfs                          3.9G     0  3.9G   0% /dev/shm
tmpfs                          3.9G  377M  3.5G  10% /run
tmpfs                          3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/mapper/rhel-root           13G   12G  1.3G  91% /
/dev/sda1                     1014M  274M  741M  27% /boot
/dev/mapper/vg_data1-lv_www1    25G   18G  8.0G  69% /var/www
/dev/mapper/vg_data1-lv_mysql  8.0G  6.5G  1.6G  81% /var/lib/mysql
/dev/mapper/vg_data1-lv_logs1  5.0G  3.4G  1.7G  68% /var/log
10.40.33.40:/var/www/html/mnt   35G   23G   13G  65% /var/www/html/mnt
tmpfs                          781M     0  781M   0% /run/user/1000

So volume /dev/mapper/rhel-root uses 91% of memory. Now, I'd like to find out where most data is in /, but only in that volume and not in all. Can I do this anyhow easily? I'd like to do something like du -sch /dev/mapper/rhel-root/*, that is / without /boot, /var/www, /var/lib/mysql and so on.

Eventually, I'd like to find potential large and unneeded (forgotten) files that takes space in the volume /dev/mapper/rhel-root, so I can delete them if possible.

Does this answer your question? [Disk full, du tells different. How to further investigate?](https://serverfault.com/questions/275206/disk-full-du-tells-different-how-to-further-investigate) — Gerald Schneider, Sep 04 '20 at 11:37

score 1 · Answer 1 · answered Oct 31 '20 at 15:44

You can do a sneaky:

mount /dev/mapper/rhel-root /mnt

find /mnt -xdev -type f -size +20M -exec ls -lh {} \;

umount /mnt

Then go through the results, of course you can change the size you are looking for.

There is also the possibility that you have something still clinging to a fd, i.e deleted a log instead of rotating it.

For this you can check:

lsof | grep deleted

Example:

[root@xx-xxxxx51:/var/log/wildfly]$ lsof | grep -i delete
S80wildfl  2224       root    1w      REG              253,3   64821642       2432 /var/log/wildfly/console.log-20170331 (deleted)
S80wildfl  2224       root    2w      REG              253,3   64822168       2432 /var/log/wildfly/console.log-20170331 (deleted)
runuser    2228       root    1w      REG              253,3   64825188       2432 /var/log/wildfly/console.log-20170331 (deleted)
runuser    2228       root    2w      REG              253,3   64825188       2432 /var/log/wildfly/console.log-20170331 (deleted)
bash       2235    wildfly    1w      REG              253,3   64825188       2432 /var/log/wildfly/console.log-20170331 (deleted)
bash       2235    wildfly    2w      REG              253,3   64825188       2432 /var/log/wildfly/console.log-20170331 (deleted)
standalon  2237    wildfly    1w      REG              253,3   64825188       2432 /var/log/wildfly/console.log-20170331 (deleted)
standalon  2237    wildfly   10w      REG              253,3   64825188       2432 /var/log/wildfly/console.log-20170331 (deleted)
java       2307    wildfly    1w      REG              253,3   64909558       2432 /var/log/wildfly/console.log-20170331 (deleted)
java       2307    wildfly    2w      REG              253,3   64909558       2432 /var/log/wildfly/console.log-20170331 (deleted)
su        19197       root    2u      CHR              136,1        0t0          4 /dev/pts/1 (deleted)
standalon 19200 terinopadm   10u      CHR              136,1        0t0          4 /dev/pts/1 (deleted)
java      19269 terinopadm    2u      CHR              136,1        0t0          4 /dev/pts/1 (deleted)

Temporary solution to recover the used space:

[root@xx-xxxxx51:/var/log/wildfly]$ lsof -p 2235
COMMAND  PID    USER   FD   TYPE DEVICE   SIZE/OFF   NODE NAME
bash    2235 wildfly  cwd    DIR  253,0       4096      2 /
bash    2235 wildfly  rtd    DIR  253,0       4096      2 /
bash    2235 wildfly  txt    REG  253,0     941880 138736 /bin/bash
bash    2235 wildfly  mem    REG  253,0     161704 131467 /lib64/ld-2.12.so
bash    2235 wildfly  mem    REG  253,0    1930416 131482 /lib64/libc-2.12.so
bash    2235 wildfly  mem    REG  253,0      23088 131492 /lib64/libdl-2.12.so
bash    2235 wildfly  mem    REG  253,0     134792 132059 /lib64/libtinfo.so.5.7
bash    2235 wildfly  mem    REG  253,0   99164480   9874 /usr/lib/locale/locale-archive
bash    2235 wildfly  mem    REG  253,0      26060   2707 /usr/lib64/gconv/gconv-modules.cache
bash    2235 wildfly    0r   CHR    1,3        0t0   3845 /dev/null
bash    2235 wildfly    1w   REG  253,3 4159791104   2432 /var/log/wildfly/console.log-20170331 (deleted)
bash    2235 wildfly    2w   REG  253,3 4159791104   2432 /var/log/wildfly/console.log-20170331 (deleted)

[root@xx-xxxxx51:/var/log/wildfly]$ ll /proc/2235/fd/2
l-wx------ 1 wildfly wildfly 64 Mar 31 08:05 /proc/2235/fd/2 -> /var/log/wildfly/console.log-20170331 (deleted)

[root@xx-xxxxx51:/var/log/wildfly]$ : > "/proc/2235/fd/2"

Note in this case, this does not solve the issue, you need to either restart your application or reboot the server.

Hope it helps.

score 0 · Answer 2 · answered Sep 04 '20 at 12:36

For me one of the below commands solves the purpose:

Go to the Root directory and run

du -k | sort -n | perl -ne 'if ( /^(\d+)\s+(.$)/){$l=log($1+.1);$m=int($l/log(1024)); printf ("%6.1f\t%s\t%25s %s\n",($1/(2**(10$m))),(("K","M","G","T","P")[$m]),""x (1.5$l),$2);}'

Top 20 in current directry and sub directory

find . -xdev -printf "%s %p\n" |sort -nr|head -20

Top 30 in /opt directory and subdirectory

du -a /opt | sort -n -r | head -n 30

Disk usage in LVM: Which directories contains most data?

2 Answers2