Cannot track down the large size file/direcotry under /root

Question

OS: RHEL 6.9

Problem
My file system /dev/mapper/vg_rayruhsso-lv_root that is mounted on / has no available size.

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_rayruhsso-lv_root
                      148G  145G     0 100% /
tmpfs                  20G     0   20G   0% /dev/shm
/dev/sda1             477M   41M  411M   9% /boot
/dev/mapper/vg_rayruhsso-lv_home
                      270G   54G  202G  22% /home

What I want to solve
Figuring out what consumed the whole space (I don't want to extend it's size, since it was extended last week from 52G to 148G). The problem is that while tracking down, I found that /root is consuming much space (128G).

# du -sh *
984K
9.3M    bin
39M     boot
4.0K    CdbsMutex
4.0K    cgroup
180K    dev
41M     etc
54G     home
215M    lib
28M     lib64
16K     lost+found
4.0K    media
0       misc
4.0K    mnt
0       net
70M     opt
du: cannot access `proc/4754/task/4754/fd/4': No such file or directory
du: cannot access `proc/4754/task/4754/fdinfo/4': No such file or directory
du: cannot access `proc/4754/fd/4': No such file or directory
du: cannot access `proc/4754/fdinfo/4': No such file or directory
0       proc
128G    root
17M     sbin
0       selinux
4.0K    srv
0       sys
5.5M    tmp
12G     u01
3.2G    usr
2.0G    var

I have checked its sub-directories and found out that the largest file/directory in there is 330M even though du -sh displays 128G.

#du -sh /root/*
4.0K    /root/anaconda-ks.cfg
4.0K    /root/Desktop
4.0K    /root/Documents
4.0K    /root/Downloads
64K     /root/install.log
12K     /root/install.log.syslog
4.0K    /root/Music
4.0K    /root/Pictures
4.0K    /root/Public
4.0K    /root/Templates
330M    /root/veritas
4.0K    /root/Videos

# cd /root/
# du -sh
128G

What's wrong? How to investigate what file is consuming all this space?

I tried to find out the open files but I didn't return any output:

lsof | grep deleted

Any idea?

Edit:

# du -h --max-depth=1 /root | sort -h
4.0K    /root/Desktop
4.0K    /root/Documents
4.0K    /root/Downloads
4.0K    /root/.gvfs
4.0K    /root/Music
4.0K    /root/.nautilus
4.0K    /root/Pictures
4.0K    /root/Public
4.0K    /root/Templates
4.0K    /root/Videos
8.0K    /root/.abrt
8.0K    /root/.ssh
12K     /root/.dbus
16K     /root/.gnote
16K     /root/.gnupg
16K     /root/.thumbnails
20K     /root/.cache
28K     /root/.gnome2
64K     /root/.config
144K    /root/.pulse
276K    /root/.gconf
364K    /root/.local
396K    /root/.kde
330M    /root/veritas
128G    /root

Use `du -h --max-depth=1 /root | sort -h` to search for your space consuming stuff. At the moment you are skipping the dot-files and dot-directories, due to glob expansion of du -sh /root/*. Please add the modified command and output to your question. — Thomas, Mar 04 '18 at 12:42
had a typo in the `sort` command`. Use `sort -h` instead of `sort -u`. — Thomas, Mar 04 '18 at 12:43
I want to send you the output but didn't due to format issue in the comment section. Anyway, all the files are under 330M — Mussa, Mar 04 '18 at 12:46
Can't you edit your question and put the new output there including the command? — Thomas, Mar 04 '18 at 12:50
Thanks for the suggestion. This is to inform you that i have included the output along with the given command in the "OP". Kindly, find it under "Edit" section. — Mussa, Mar 04 '18 at 12:54
Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/74014/discussion-between-thomas-and-mussa). — Thomas, Mar 04 '18 at 13:00

Thomas · Accepted Answer · 2018-03-04T13:17:17.933

1

After a bit forth and back, and a private discussion...

Use du -h --max-depth=1 /root | sort -h to search for your space consuming stuff.

At the moment you are skipping the dot-files and dot-directories, due to glob expansion of du -sh /root/*.

Now with the results including the dot-files and dot-folders from the edit, it seems the files consuming the space are directly in /root.
To get an overview of which files take most space in /root use the command as follows.

ls -alhSr /root

edited Mar 04 '18 at 13:17

answered Mar 04 '18 at 12:31

Thomas

4,225
5
23
28

Hi Thomas! Thanks for giving the time to answer. I have run that command and the largest file is 396M. – Mussa Mar 04 '18 at 12:38
It doesn't have to be an actual file, it can be a bound file descriptor, that has the storage allocated and never released. Check what processes are using /root and try to restart them one by one, until you release the used storage. – Gothrek Mar 05 '18 at 11:35
@Gothrek: If it would have been that filedescriptor, the space would not have been seen with `du` and would only have been reflected by `df`. Also the OP ruled out open filedescriptors. – Thomas Mar 05 '18 at 18:13
Oh, you’re right – Gothrek Mar 06 '18 at 07:15

score 0 · Answer 2 · answered Mar 04 '18 at 13:59

Is it possible that you've issued a delete command and it never ended because the files you were trying to delete were in use by some application?

Try running:

/usr/bin/lsof | grep deleted

The above command will display all the files which are waiting to be deleted and are being used by a process.

If such files exist, kill their process and check the capacity again.

Explanation:

On Linux or Unix systems, deleting a file via rm or through a file manager application will unlink the file from the file system's directory structure; however, if the file is still open (in use by a running process) it will still be accessible to this process and will continue to occupy space on disk. Therefore such processes may need to be restarted before that file's space will be cleared up on the filesystem.

More information on this subject can be found here.

score 0 · Answer 3 · answered Mar 05 '18 at 09:56

0

You might also have a look at ncdu. It's a lot like treesize and helped me many times to locate large files and directory structures.

answered Mar 05 '18 at 09:56

M1k3y

19
5

Cannot track down the large size file/direcotry under /root

3 Answers3