Diffrence between real and showed by docker system df disc usage

Question

I have docker in version 18.03.1-ce that support command docker system df. His output:

Images space usage:

REPOSITORY                               TAG                 IMAGE ID            CREATED ago             SIZE                SHARED SIZE         UNIQUE SiZE         CONTAINERS
registry.gitlab.com/precisesale/app      latest              b7833546c2cf        About an hour ago ago   252.1MB             123.8MB             128.4MB             1
healthdiary/app                          latest              565c6d3906e6        2 days ago ago          312.2MB             123.8MB             188.4MB             1
mongo                                    latest              f93ff881751f        5 days ago ago          367.6MB             0B                  367.6MB             2
nginx                                    latest              b175e7467d66        6 weeks ago ago         108.9MB             0B                  108.9MB             1
jwilder/docker-gen                       latest              8959ee34c769        2 months ago ago        19.91MB             4.148MB             15.77MB             1
jrcs/letsencrypt-nginx-proxy-companion   latest              17939ceb7a52        2 months ago ago        86.86MB             4.148MB             82.71MB             1

Containers space usage:

CONTAINER ID        IMAGE                                    COMMAND                  LOCAL VOLUMES       SIZE                CREATED ago         STATUS              NAMES
c20dc3438552        healthdiary/app                          "./entrypoint.sh nod…"   0                   0B                  8 minutes ago ago   Up 8 minutes        healthdiary_app_1
bf8c4307dcbb        mongo:latest                             "docker-entrypoint.s…"   1                   0B                  8 minutes ago ago   Up 8 minutes        healthdiary_mongo_1
47fced8d18fe        registry.gitlab.com/precisesale/app      "./entrypoint.sh nod…"   0                   0B                  9 minutes ago ago   Up 9 minutes        precisesale_app_1
597d97d5c1fa        mongo:latest                             "docker-entrypoint.s…"   1                   0B                  9 minutes ago ago   Up 9 minutes        precisesale_db_1
b5bb14faa910        jwilder/docker-gen                       "/usr/local/bin/dock…"   0                   0B                  7 hours ago ago     Up 19 minutes       nginx-gen
8eee2bee084a        nginx                                    "nginx -g 'daemon of…"   0                   2B                  7 hours ago ago     Up 19 minutes       nginx-web
6b8b0cd5d938        jrcs/letsencrypt-nginx-proxy-companion   "/bin/bash /app/entr…"   0                   1.66kB              7 hours ago ago     Up 19 minutes       nginx-letsencrypt

Local Volumes space usage:

VOLUME NAME                                                        LINKS               SIZE
0a40fac6ca98e776dad972c8193362a51a485b3305979e58996545d97310a3c7   1                   0B
929b0b88849ad4d390efd4666e6a0e5f82e0e6dd34f7a09f609de90b190e6148   1                   0B

Build cache usage: 0B

Even if I do not take into account savings from shared space from two first containers summary size is 1147.5 MB

But if I measure size of docker overlay2 on disc by du I get

du -hs /var/lib/docker/overlay2/
2.7G    /var/lib/docker/overlay2/

Where is reason of difference in size of containers measured by docker system df and du?

Answer 1

I was wondering the same thing some time ago. It’s not a bug, it’s a feature :-)

du -sh /var/lib/docker/overlay2

is not showing objective value because merge folders have been mounted using overlay driver and du output is not actual disk allocation size.

You can see the actual disk allocation size by examining only diff folders like:

du -shc /var/lib/docker/overlay2/*/diff

You can test this in your environment like this: run

df -h /dev/sd*
du -shc /var/lib/docker/overlay2/*/diff
du -sh /var/lib/docker/overlay2

Now start 20 centos containers and observe what has change:

for i in {1..20}; do docker run -itd centos bash; done
df -h /dev/sd*
du -shc /var/lib/docker/overlay2/*/diff
du -sh /var/lib/docker/overlay2

You can see that the actual disk allocation (df command) is just cca 200MB more than before, but “du” on whole folder outputs 4.2G allocation. “du” on “diff” folders shows 212M what is correct.

This is how Docker works and what makes it great!

Answer 2

docker root folder may contain files from other folders, such as mounted to merged, my condition is using -x of du will exclude those files out, it's amazing:

# du -sh
11G     .
# du -shx
6.5G    .
# pwd
/var/lib/docker

man du
       -x, --one-file-system
              skip directories on different file systems

overlay2# docker inspect --format "{{json .GraphDriver.Data}}" cucontainer
{"LowerDir":"/var/lib/docker/overlay2/139a5d81f92c89bd6cb25401668d0cc8b3c1e76f60e501bcc7ef8b617d5a8cde-init/diff:/var/lib/docker/overlay2/8d430789d386bdcfb9a6d9536182c35df51dfefc92ec09df17db3f558f1e7742/diff:/var/lib/docker/overlay2/21a0d25bb6909e6f7197fcbaeed29d498e6e2c3ec73bad2b89f0edbb28a47139/diff:/var/lib/docker/overlay2/65ddfba293980c1a1d52628fa9a6e5f2bb4519ab290aec4ee4e358bc3d32afa4/diff:/var/lib/docker/overlay2/fa7931a91ef7a202b04fbbc0063f9ad102ade4be12171ea6d227a19d63db169d/diff:/var/lib/docker/overlay2/093fc4396dccff75b0b17b48dbe0051bde0f7a98f8c23d4d11cc7046554f829a/diff:/var/lib/docker/overlay2/d6afcf2f8dcbe66decc85a1cc8a1557481ca833f399219ab358fa0fc15a89b9e/diff:/var/lib/docker/overlay2/5c511f7c22bcdd40d639574662abfb3a7fbcec89e908ddec5affe351e5dca7c9/diff:/var/lib/docker/overlay2/46d9f1129dd57ec5a1444325ea7b0fb6174b016736658372a13fd41d6391934c/diff:/var/lib/docker/overlay2/54f2b67cdb11defedfc5774f91bdf861318a6811865d76ee0724b18ab7302b4f/diff:/var/lib/docker/overlay2/b3cc77bd18c649b29f1e9bcc4da4c98e4d3a23584deea7633a65e66120c74e3f/diff:/var/lib/docker/overlay2/15d61ed3cc41f452daf62e2d70e623ed450bb73b267684734e9185adf16d02f5/diff:/var/lib/docker/overlay2/d80154af0de3813db13eb38871790c22766ccbfd4704fd7de0c3d3da8421b2b1/diff:/var/lib/docker/overlay2/827c202d462efdea44a10f0b679681a74a8d616cc65b53565978c382929bc54b/diff:/var/lib/docker/overlay2/ca7b2b38a0f24dfddf4279f7166170b282c0ee50a90fa4afc5268fd132fd2d12/diff:/var/lib/docker/overlay2/e70f654ea5e956f3952c9b60cbfc07be37d65a2e6d266cd5758d5f7fada77b80/diff:/var/lib/docker/overlay2/e655cb9ba7e783d7e78a739714f9843cfd3243c0dd44cf91183b35df90337163/diff","MergedDir":"/var/lib/docker/overlay2/139a5d81f92c89bd6cb25401668d0cc8b3c1e76f60e501bcc7ef8b617d5a8cde/merged","UpperDir":"/var/lib/docker/overlay2/139a5d81f92c89bd6cb25401668d0cc8b3c1e76f60e501bcc7ef8b617d5a8cde/diff","WorkDir":"/var/lib/docker/overlay2/139a5d81f92c89bd6cb25401668d0cc8b3c1e76f60e501bcc7ef8b617d5a8cde/work"}
overlay2# du -sh 139a5d81f92c89bd6cb25401668d0cc8b3c1e76f60e501bcc7ef8b617d5a8cde
1.5G    139a5d81f92c89bd6cb25401668d0cc8b3c1e76f60e501bcc7ef8b617d5a8cde
overlay2# du -xsh 139a5d81f92c89bd6cb25401668d0cc8b3c1e76f60e501bcc7ef8b617d5a8cde
8.5M    139a5d81f92c89bd6cb25401668d0cc8b3c1e76f60e501bcc7ef8b617d5a8cde
overlay2# mount | grep 139a5
overlay on /var/lib/docker/overlay2/139a5d81f92c89bd6cb25401668d0cc8b3c1e76f60e501bcc7ef8b617d5a8cde/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/B5YSAXFCGTY44CYSH3AANFEODQ:/var/lib/docker/overlay2/l/VK5DCEX7S3A7AVCWFZELC7WVVJ:/var/lib/docker/overlay2/l/KZ4CE35HG2GKUO6QFQ7XAQMWCR:/var/lib/docker/overlay2/l/SL23EZUQQVRKRWPKTUHEKZB3CF:/var/lib/docker/overlay2/l/G663PGO2HQK4M6O2NZLCTZGOIX:/var/lib/docker/overlay2/l/52IHQGC4TJPSCRHB33ITEFVSDU:/var/lib/docker/overlay2/l/7V6AA72MMV4ZAWD3LEUMQGVF3X:/var/lib/docker/overlay2/l/TQYFM75LMGDMF5NBRKIHNDUVBG:/var/lib/docker/overlay2/l/BC7S7SI3FI63HBFATXWT4SCCTZ:/var/lib/docker/overlay2/l/CA4EFFDQHFAPBILGHPL4DW7DEL:/var/lib/docker/overlay2/l/CEXD6LSJETQXOITYVIIZSYUQFV:/var/lib/docker/overlay2/l/EZORVN3Z4AAZCW4EQJ6ZIC3DPF:/var/lib/docker/overlay2/l/OJAB45D73WG2ONCNKB6R6BOA5N:/var/lib/docker/overlay2/l/44Z6LRZUBYM5ARXNKVKFKWWWXE:/var/lib/docker/overlay2/l/UXMEMGQ3N4KAXEZ52RJQVHNZNX:/var/lib/docker/overlay2/l/M75YFIRTKBWLIQVV7JE6MU4K3E:/var/lib/docker/overlay2/l/4U25BGTQQ466TVIQG67G6GFV4I,upperdir=/var/lib/docker/overlay2/139a5d81f92c89bd6cb25401668d0cc8b3c1e76f60e501bcc7ef8b617d5a8cde/diff,workdir=/var/lib/docker/overlay2/139a5d81f92c89bd6cb25401668d0cc8b3c1e76f60e501bcc7ef8b617d5a8cde/work)
overlay2#