I have a Linux machine (Ubuntu with Btrfs) which runs Docker and 4 cointainers.
Docker images
# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
alpine 3.8 11cd0b38bc3c 9 months ago 4.41MB
grafana/grafana 5.1.5 6216a7d47b26 10 months ago 238MB
quay.io/prometheus/prometheus v2.3.1 b82ef1f3aa07 10 months ago 119MB
Docker containers
# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e7fdc76b0437 alpine:3.8 "/alertmanager/alert�~@�" 8 days ago Up 12 minutes 0.0.0.0:9093-9094->9093-9094/tcp monitoring_alert_manager_1
732ce2eaace7 grafana/grafana:5.1.5 "/run.sh" 8 days ago Up 12 minutes 0.0.0.0:3000->3000/tcp monitoring_grafana_1
75587d6787e4 quay.io/prometheus/prometheus:v2.3.1 "/bin/prometheus --l�~@�" 8 days ago Up 12 minutes 0.0.0.0:9090->9090/tcp monitoring_prometheus_1
a1d57121c8ce alpine:3.8 "/blackbox_exporter �~@�" 6 months ago Up 12 minutes 0.0.0.0:9115->9115/tcp monitoring_blackbox_exporter_1
Volumes
# du -h -d 1 "/var/lib/docker/volumes/"
451M /var/lib/docker/volumes/monitoring_prometheus
2.3M /var/lib/docker/volumes/monitoring_grafana
4.0K /var/lib/docker/volumes/monitoring_alertmanager
453M /var/lib/docker/volumes/
The problem is that I cannot explain what is using much of the disk space.
# btrfs fi df /
Data, single: total=13.01GiB, used=10.82GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=639.75MiB, used=101.23MiB
GlobalReserve, single: total=16.00MiB, used=0.00B
So, around 11GB is used (also reported by df -h), but I would expect that much less should be used. There are 3 images, which take around 350MB in total and around 450MB is used for actual data, so less than 1GB + OS itself. If I try to dig where are the files that are taking up disk space - du reports that only 4.8G of the disk is used (ignoring btrfs' copy on write and 2.6G of this is /var/lib/docker).
# du -sh /
du: cannot access '/proc/1209/task/1209/fd/4': No such file or directory
du: cannot access '/proc/1209/task/1209/fdinfo/4': No such file or directory
du: cannot access '/proc/1209/fd/3': No such file or directory
du: cannot access '/proc/1209/fdinfo/3': No such file or directory
4.8G /
I found some hints that this can be in case that some large files are open by application, but deleted. I looked in lsof and rebooted the machine, but the result is the same. Second thing could be subvolumes, but it's apparently the root one that is taking majority of disk space.
# btrfs-du /
Subvolume Total Exclusive ID
var/lib/docker/btrfs/subvolumes/2fddeb2724f7b7ebd816c313e67580b71f36a07386bdb126e3b269cd25d40081 1.20MiB 64.00KiB 259
var/lib/docker/btrfs/subvolumes/5652e05094bbf4b171053cc6be6626d922405f699d79be265f8487b8ce3fd643 3.93MiB 784.00KiB 260
var/lib/docker/btrfs/subvolumes/c9b5313b82c7aa1e562e681e5c3344ac770e2e69820ee6bdbe9b14bf654e472f 71.81MiB 784.00KiB 261
var/lib/docker/btrfs/subvolumes/cf656f2dd19233625e66974ca89fd295583526daf4614e3f738458f629aaf8b6 114.72MiB 784.00KiB 262
var/lib/docker/btrfs/subvolumes/93a4c2f43dc245d01715a9b3c46f268f319e2d6dba2c56a7ad6078c1a98775d9 114.74MiB 784.00KiB 263
var/lib/docker/btrfs/subvolumes/5237ab02149b30f6b45de51cee89a9bec76cdc26b2e40d3cd2e37a88b331fbae 114.76MiB 800.00KiB 264
var/lib/docker/btrfs/subvolumes/6726bf1b2712fb2ed11db8629c339fd0f0afad55a9655a5e3d6ae7ca809e4c62 114.79MiB 800.00KiB 265
var/lib/docker/btrfs/subvolumes/b4eed058bca3d31d9f118b5db82515a3aa6dabcff0fb7d564e2bda1462459ce9 114.79MiB 800.00KiB 266
var/lib/docker/btrfs/subvolumes/76a59d8518448c251436254b0f7c206f65d3ccc56e3e45d1ed571c0d9377618d 114.79MiB 800.00KiB 267
var/lib/docker/btrfs/subvolumes/c740a1bfe14870a61e7fe26da24f0279a1df1d658bcfa4cf537d355f3e7c0c83 57.26MiB 3.64MiB 268
var/lib/docker/btrfs/subvolumes/23c7fccc98ddc4aa5fa1c14f54ae26efcd667e3f7a57fdf4b0870369c85861f4 232.76MiB 3.14MiB 269
var/lib/docker/btrfs/subvolumes/fe33b8bde93cb411035756fa901091261201b86260501035084db889f3e353f4 232.78MiB 3.14MiB 270
var/lib/docker/btrfs/subvolumes/300d2b8c2dfcbbccd09d3cae14d0d80618f5ccd2b53ddacab110057bc4cf9c4f 4.47MiB 160.00KiB 322
var/lib/docker/btrfs/subvolumes/7b5e78985035cc9b7e9c5acdda898cfa661dc0aa85809aa9a2fa437df43a38f9-init 4.47MiB 160.00KiB 509
var/lib/docker/btrfs/subvolumes/7b5e78985035cc9b7e9c5acdda898cfa661dc0aa85809aa9a2fa437df43a38f9 4.47MiB 160.00KiB 510
var/lib/docker/btrfs/subvolumes/2f34bf0d5958d5d156970742d13859f82ec5990fb4e610e1155b95fb1483f3b5-init 114.79MiB 800.00KiB 699
var/lib/docker/btrfs/subvolumes/2f34bf0d5958d5d156970742d13859f82ec5990fb4e610e1155b95fb1483f3b5 114.79MiB 800.00KiB 700
var/lib/docker/btrfs/subvolumes/e202293240ca4a4c1f6b996be9ba2d044d4b3007df0c7e5a29a32a949f49ae57-init 232.78MiB 3.14MiB 701
var/lib/docker/btrfs/subvolumes/e202293240ca4a4c1f6b996be9ba2d044d4b3007df0c7e5a29a32a949f49ae57 232.78MiB 3.76MiB 702
var/lib/docker/btrfs/subvolumes/ffb1461bb86799876e58e4213483e3da2b1ffee401e975087e296015d6591972-init 4.47MiB 160.00KiB 703
var/lib/docker/btrfs/subvolumes/ffb1461bb86799876e58e4213483e3da2b1ffee401e975087e296015d6591972 4.47MiB 160.00KiB 704
Total exclusive data 25.42MiB
Quotas
# btrfs quota rescan /
quota rescan started
# btrfs qgroup show /
qgroupid rfer excl
-------- ---- ----
0/5 10.80GiB 10.80GiB
0/259 1.21MiB 64.00KiB
0/260 3.94MiB 784.00KiB
0/261 71.81MiB 784.00KiB
0/262 114.73MiB 784.00KiB
0/263 114.74MiB 784.00KiB
0/264 114.77MiB 800.00KiB
0/265 114.79MiB 800.00KiB
0/266 114.79MiB 800.00KiB
0/267 114.79MiB 800.00KiB
0/268 57.27MiB 3.64MiB
0/269 232.77MiB 3.14MiB
0/270 232.79MiB 3.14MiB
0/322 4.47MiB 160.00KiB
0/509 4.47MiB 160.00KiB
0/510 4.47MiB 160.00KiB
0/699 114.79MiB 800.00KiB
0/700 114.79MiB 800.00KiB
0/701 232.79MiB 3.14MiB
0/702 232.79MiB 3.77MiB
0/703 4.47MiB 160.00KiB
0/704 4.47MiB 160.00KiB
Does anyone have any clue what is using disk space?
