8

So im a very new sysadmin, just got out of school and doing my internship. Only problem is that im the only sysadmin in the place and no one to show me the job. Anyway, it's a very small company, one CentOs server with that configuration :

Filesystem            Size  Used Avail Use% Mounted on

/dev/sda3             184G  140G   35G  81% /

tmpfs                 2.3G     0  2.3G   0% /lib/init/rw

udev                  2.3G  212K  2.3G   1% /dev

tmpfs                 2.3G     0  2.3G   0% /dev/shm

/dev/sda1             4.6G  156M  4.2G   4% /boot

/dev/sda4              33G  176M   31G   1% /tmp

/dev/sdb1             1.8T  1.8T     0 100% /media/backupInterne

/dev/sdd1             917G  470G  401G  54% /media/Data

I got here only a few days ago, and noticed the full disk right away and im working on fixing that problem. My other problem here is sda3 now at 81%. 4 days ago, it was at 79%.

I ran the du -ah | sort -rh command on the / root directory, nothing stands out. Did it with a few days appart since the sda3 partition is filling up quickly, no major differences that could explain why its growing.

Thanks a lot

Nathan C
  • 15,059
  • 4
  • 43
  • 62
littleadmin
  • 177
  • 1
  • 1
  • 11
  • 4
    My guess would be log growth because `sdb1` is full, but we'll see how big `/var` is.. what do you get from `du -sh /*`? – Shane Madden Dec 06 '13 at 16:55
  • 1
    If you want to see which files have chaged, then you can use [`find`](http://linux.die.net/man/1/find) with -mtime n[smhdw]. I suspect that Shane is right. Part of it may be the log files complaining about the full sdb1 volume. The command could look like this: `find / -type f -mtime 1d -print` If your find supports `--exclude-dir=` then you mioght want to exclude /dev and /proc. – Hennes Dec 06 '13 at 17:20
  • /var is 1.7G and the size of it barely moved in the past few days, it was my first idea to check that. I always run the du command with an --exclude='media' since there's nothing in that directory then mounted directories – littleadmin Dec 06 '13 at 17:26
  • Since you wrote that you are a new admin I would to point to one of the most common reasons of growing disk usage. Log files. If you open a file (e.g. the log from a webserver) and later delete that file then the file will still use disk space until the program closes its handle to that file. This last is sometimes solved by sending a signup (`kill -1 PID` -> reread config file and restart for many deamons) or by the blunt axe of rebooting. – Hennes Dec 06 '13 at 17:26
  • In case it is log growth, it may settle in a few days when weekly log rotation kicks in. – ptman Dec 06 '13 at 17:57
  • Also remember to check how you're doing inode-wise: `df -i`. Often I run out of inodes before running out of space if I just check space usage with `du` – ptman Dec 06 '13 at 17:58
  • " I always use ... --exclude=media" - may be this is the root of the problem, you can not see casual "media" subdirs. Use -x option to stay in one filesystem instead. – Veniamin Dec 08 '13 at 17:18
  • I thought about that, but all the directories we have in /media are mounted point. Theres no files either. I keep comparing the size of the logs, but they stay pretty much the same. No errors messages either. Im lost, and the space is running out quickly – littleadmin Dec 09 '13 at 18:01

3 Answers3

6

Here is what I use in trying to figure out problems like this. 

du -s `ls -a | egrep -v '\.\.'` | sort -nr | head

It will show you the usage per directory/file in the current directory. From there you step down into sub dirs until you find something obvious.

Having everything in one big partition can make diagnosing problems like this difficult. Another approach to try is using 

lsof

to see what files are open by the various process and see if you can find some clues. This is very hit or miss though.

Fred the Magic Wonder Dog
  • 2,095
  • 12
  • 17
3

It sounds a lot like a similar issue I have all the time with deleted files (but the reference is still there).

If we're talking a Linux system, run:

lsof +L1

This will be a list of deletes files, but are still open and being used by something. The key is to get whatever has the filehandle open to release it.

Eirik Toft
  • 834
  • 9
  • 20
  • Sadly didn't give any files that could explain the loss of space that is happening everyday. thanks – littleadmin Dec 06 '13 at 21:32
  • Is it possible that something is writing to a directory that is then getting mounted over with a filesystem? I had gigabytes of unexplained hair-pulling over that once. While this should never be possible, I was able to prove it can happen. – Eirik Toft Dec 06 '13 at 22:32
  • Im thinking of something like that too, but how could that happen? And how do i verify it without umounting everything? – littleadmin Dec 09 '13 at 18:05
2

I finally figured out what was going on. One of the mounted point wasnt mounting correctly and therefor was doing the backup directly on sda3.

Thanks everyone for the help

littleadmin
  • 177
  • 1
  • 1
  • 11