AWS /dev/xvda1 directory is full

Question

Sometimes the command df -h gives /dev/xvda1 directory full. But not all the time. I have checked here & here and tried the solutions mentioned there but that didn't help.

Can someone guide what can be the cause?

Answer 1

Find a folder or files containing much volume of disk with

du -hsc *

this command will give you result that which folder is containing much data.

Note:- run this command as a root user.

Answer 2

If your drive is "randomly" filling up, a common explanation would be that you have some form of logs that are dumping tons of information, and logrotate is cleaning them up on a routine basis. To identify everything that logrotate would be rotating, look at the /etc/logrotate.conf file and the /etc/logrotate.d directory.

I usually refer to this article or this article when I need to find large files or directories on my file system. You may need to locate large files, or large directories, exclusively. This is because the large volume of data that is being written to your disk may not be in the form of one large file, but thousands of tiny ones.

Find top 20 largest directories on the / drive (note, you may need to expand this past the top 20, due to /var, /var/log and /var/log/httpd all being returned if /var/log/httpd is very large):

sudo du -a / | sort -n -r | head -n 20

Find top 20 largest files on the / drive :

sudo find / -type f -printf "%s\t%p\n" | sort -n | tail -20

One other suggestion is to look for where your /home is mounted. Does it have its own partition, or is it being served off of the / partition? If the latter, then someone or some service could inadvertently be uploading large files to their home directory.

If the above has not solved your issue, can you please elaborate on the scenario when df -h reports the drive IS NOT full? How full is it when it's not full? Can you draw any correlation to time of day, day of week, current work load, etc. for when the drive reports full vs when it does not report full?

Answer 3

I also recommend ncdu, a pretty cool tool that is much more informative than the original du. I've been using it for many years.

When you run ncdu, the output is sorted by the size; then its repl allows you to navigate the directory tree, and see, which subdirectories of a directory are the largest. Gives you a clearer picture of a problem, compared to du.