Why does docker say my container's disk is full when it's not (QubesOS)?

Question

Why does a fresh container on a small-ish docker image say that its disk is full of 10G when it's not?

I'm running this in a Debian 10 AppVM in QubesOS. In Debian 10, I do:

sudo apt-get -y install docker.io
sudo docker pull node:13-buster-slim

At the time of writing, this gives me docker v18.09.1 using the 'overlay2' storage driver by default.

root@coviz:~# sudo docker --version
Docker version 18.09.1, build 4c52b90
root@coviz:~# docker info | grep Storage
Storage Driver: overlay2
root@coviz:~# 

My docker host now has only this 181M docker image and no containers. The docker host is using only 0.5G out of 20G available. Plenty of free space.

root@coviz:~# sudo docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
root@coviz:~# sudo docker image ls -a
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
node                13-buster-slim      e4217af9b7c7        9 days ago          181MB
root@coviz:~# sudo docker system df
TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              1                   0                   180.7MB             180.7MB (100%)
Containers          0                   0                   0B                  0B
Local Volumes       0                   0                   0B                  0B
Build Cache         0                   0                   0B                  0B
root@coviz:~# 

I'm working on creating a Dockerfile for my project, so I execute the following command to spin up a new container from the above base image and drop me in a shell on that temporary container

root@coviz:~# docker run --rm -it --entrypoint /bin/bash e4217af9b7c7
root@97a318c599ab:/#

It isn't long before I encounter issues when testing out commands to install dependencies with apt-get. I think the issue is that apt needs to store cache data to /var/lib/apt/lists/. The actual error is at least one invalid signature was encountered, but it actually appears to be a disk fill issue (the apt key verification fails because it can't store the signature to disk). Running an apt-clean doesn't help; it's already empty. This is a fresh container based on a fresh image.

Checking the disk with df in this fresh container immediately shows that there's only 17M of disk space available, but I can only account for ~200M with du. Again, this is a fresh container, so I highly doubt this is an issue with a file stuck in a 'deleting' state still opened by a process.

root@97a318c599ab:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay         9.6G  9.1G   17M 100% /
tmpfs            64M     0   64M   0% /dev
tmpfs           255M     0  255M   0% /sys/fs/cgroup
/dev/xvda3      9.6G  9.1G   17M 100% /etc/hosts
shm              64M     0   64M   0% /dev/shm
tmpfs           285M     0  285M   0% /proc/asound
tmpfs           285M     0  285M   0% /proc/acpi
tmpfs           285M     0  285M   0% /proc/scsi
tmpfs           285M     0  285M   0% /sys/firmware
root@97a318c599ab:/# du -sh /*
4.8M    /bin
4.0K    /boot
0   /dev
612K    /etc
20K /home
12M /lib
4.0K    /lib64
4.0K    /media
4.0K    /mnt
5.2M    /opt
du: cannot access '/proc/11/task/11/fd/4': No such file or directory
du: cannot access '/proc/11/task/11/fdinfo/4': No such file or directory
du: cannot access '/proc/11/fd/4': No such file or directory
du: cannot access '/proc/11/fdinfo/4': No such file or directory
0   /proc
136K    /root
8.0K    /run
4.1M    /sbin
4.0K    /srv
0   /sys
2.2M    /tmp
160M    /usr
5.9M    /var
root@97a318c599ab:/# 

Moreover, docker ps -s shows that the size of the "writable layer" on my container is empty (0B):

root@coviz:~# docker ps -a -s
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES               SIZE
320af1498086        e4217af9b7c7        "docker-entrypoint.s…"   15 minutes ago      Up 15 minutes                           epic_leakey         0B (virtual 181MB)
root@coviz:~# 

So why is this fresh docker container's disk (based on a ~200M image) full? What's taking-up those ~9G of unaccounted space?

Answer 1

This is an issue with the way QubesOS also layers its disk space for AppVMs.

This was the disk space on the docker host (the Debian 10 AppVM). Note that its 20G of free disk space is in /rw/ but that the available disk space mounted to / is the same as what was reported inside the container.

user@coviz:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda3      9.6G  9.1G   17M 100% /
none            9.6G  9.1G   17M 100% /usr/lib/modules
devtmpfs        1.9G     0  1.9G   0% /dev
tmpfs           1.0G     0  1.0G   0% /dev/shm
tmpfs           777M   17M  760M   3% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           776M     0  776M   0% /sys/fs/cgroup
/dev/xvdb        20G  418M   20G   3% /rw
tmpfs            32M   12K   32M   1% /run/user/1000
user@coviz:~$ 

The solution was to add a config to /rw/config/qubes-bind-dirs.d/ that binds the docker dirs '/var/lib/docker and /etc/docker into /rw/. Not only does this actually give the docker containers access to Qubes's bigger "Private storage", but it also makes that storage persistent across reboots.

The above can be achieved by executing the following on your Debian 10 AppVM:

sudo mkdir -p /usr/lib/qubes-bind-dirs.d
sudo cat << EOF > /usr/lib/qubes-bind-dirs.d/50_user.conf
binds+=( '/var/lib/docker' )
binds+=( '/etc/docker' )
binds+=( '/root/.docker' )
EOF

On older versions of QubesOS (< r4.0), the qubes-bind-dirs.d is in /rw/config/ ^[1]:

sudo mkdir -p /rw/config/qubes-bind-dirs.d
sudo cat << EOF > /rw/config/qubes-bind-dirs.d/50_user.conf
binds+=( '/var/lib/docker' )
binds+=( '/etc/docker' )
binds+=( '/root/.docker' )
EOF

And then reboot the AppVM. When it comes back up, it will be like this:

root@coviz:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda3      9.6G  8.9G  217M  98% /
none            9.6G  8.9G  217M  98% /usr/lib/modules
devtmpfs        1.9G     0  1.9G   0% /dev
tmpfs           1.0G     0  1.0G   0% /dev/shm
tmpfs           777M   17M  760M   3% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           776M     0  776M   0% /sys/fs/cgroup
/dev/xvdb        20G  617M   20G   4% /rw
tmpfs            32M   12K   32M   1% /run/user/1000
root@coviz:~# docker run --rm -it --entrypoint /bin/bash e4217af9b7c7
root@66766a23ca1b:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          20G  618M   20G   4% /
tmpfs            64M     0   64M   0% /dev
tmpfs           246M     0  246M   0% /sys/fs/cgroup
/dev/xvdb        20G  618M   20G   4% /etc/hosts
shm              64M     0   64M   0% /dev/shm
tmpfs           274M     0  274M   0% /proc/asound
tmpfs           274M     0  274M   0% /proc/acpi
tmpfs           274M     0  274M   0% /proc/scsi
tmpfs           274M     0  274M   0% /sys/firmware
root@66766a23ca1b:/#