0

I have a server running on which many docker containers run and has capacity of around 400Gb and used storage is at 290GB and usage is increasing .Disc space is mounted on "/".

Here is the output of "df -h" running as a root:

tmpfs           6.0G   71M  6.0G   2% /run
/dev/xvda1      392G  295G   88G  78% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            394G  295G   84G  78% 
/var/lib/docker/aufs/mnt/a8c1cc295d615ac7c24ae570f0e6a0872587b767f61d9261560a55a3b6c900ed
none            394G  295G   84G  78% /var/lib/docker/aufs/mnt/ec3e8f485e970a92400d7d36d4ca1b18638a1d1ecbbe65162ba4666e75358ec6

Here is the output of sudo du -h / | grep -P '^[0-9.]+G'

1.1G    /var/lib/docker/aufs/mnt/8d2bc80c9409a84a2b8cea5e35f55ad4d2c6b8f9937db3f23c988c09bd1c0e64
1.1G      /var/lib/docker/aufs/mnt/68f8312eb34763bcaf7833215adf9974c3e812b9cff6c668fd131cd0c4c2f1ff
1.9G /var/lib/docker/aufs/mnt/0e14789b512e25d900bf2140d68e0e4e82842be2645fdb3c4ffad402d1fb0260
1.2G    /var/lib/docker/aufs/mnt/1ca8b96421d90ea310cad9f2b41b116ade03f65fd61666018849dcab470bc54f
1.4G    /var/lib/docker/aufs/mnt/4e8bbe5c78cddb019f1393e97a3784f6a2083779c43ba05f5bff1daf4c9a6f13
28G /var/lib/docker/aufs/mnt
22G /var/lib/docker/aufs/diff
49G /var/lib/docker/aufs
50G /var/lib/docker
50G /var/lib
50G

Question:

1.I am not sure on running "df -h" instead of only showing "/" mount why It is showing /var/lib/.... as a mount point too.

2.On running "sudo du -h / | grep -P '^[0-9.]+G' " why it is only showing up the disc consumption upto 50G but not whole disc consumption even when I am running du against a mount point shown by a df

Output of du -hs /var/lib/*

 8.0K   /var/lib/apparmor 
 127M   /var/lib/apt
 4.0K   /var/lib/aptitude
 50G    /var/lib/docker

 removed other entries because they were very small in size

Update:When I remove all the running containers then disc space utilization comes down to 7%.Which makes mw think that there is a rogue container which is filling up space because it takes almost 10hr to raise disc space to 80%.

What steps I can take to find out the rogue container?

mount :

 /dev/xvda1     ext4     412704840 312914896  82913940  80% /
 none           aufs     412704840 312914896  82913940  80% /var/lib/docker/aufs/mnt/a8c1cc295d615ac7c24ae570f0e6a0872587b767f61d9261560a55a3b6c900ed
immrsteel
  • 103
  • 3
  • Please show the output of `du -hs /var/lib/*` – shodanshok Aug 12 '17 at 17:50
  • @shodanshok I added the output – immrsteel Aug 12 '17 at 18:04
  • Whats the output of the `mount` command also what is the content of the `/etc/fstab` It looks like the docker stuff is some sort of bind mount or loop back mount. – James Aug 12 '17 at 20:08
  • I think for #1, you actually want to use `sudo du -h --max-depth 1` (no grep afterwards). Otherwise, you get every subdirectory's size, too. Also, what file system is your root using? If it's something with snapshots, that could be taking up the space (especially with Docker!) – lungj Aug 13 '17 at 05:48
  • @James added the output – immrsteel Aug 13 '17 at 15:59
  • @lungj added the output – immrsteel Aug 13 '17 at 16:00
  • Protip: use `sudo du -sh /* | sort --human-numeric` to show a summary (`-sh`) of everything under / (`/*`), and sort it nicely. Just FYI. – shearn89 Aug 14 '17 at 10:06

1 Answers1

1

I answered on same question couple of time ago. Try to execute lsof | grep delete, to find deleted files that haven't seen by du, but use disk space. Also you could find process PID who lock this files.

Alexander Tolkachev
  • 4,608
  • 3
  • 14
  • 23
  • @ Alexander Tolkachev Thx Yeah that worked.Found out the PID using lsof command and from there container which was filling up tmp folder. BTW what is delete files which showed up in lsof represent? – immrsteel Aug 13 '17 at 16:40
  • @immrsteel see this [question](https://stackoverflow.com/questions/18200701/lsof-shows-a-file-as-deleted-but-i-can-still-see-it-in-file-system) – Alexander Tolkachev Aug 13 '17 at 16:43