List Kubernetes containers that use docker local volumes

Question

When deploying a workload that has a VOLUME in a Dockerfile, that volume may not be mapped to a persistent volume (PV/PVC) in Kubernetes.

Actually, unless a Kubernetes volume is attached to that workload, The docker daemon container will temporarily create a docker-volume when starting the container (driver type: local). Kubernetes won't be aware of it. See: are VOLUME in Dockerfile persistent in kubernetes. This docker volume will be destroyed when the pod is removed or redeployed.

It is certainly good practice to use a kubernetes volume, even ephermeal volumes (or generic ephemeral volumes... still in alpha in 1.19)

Q: How to list pods/containers that use such local volumes?

This is really important since restarting the workload/deployment/stateful-set will cause disruption (lost of ephemeral volume).

What are you deploying, that needs persistent data, that doesn't call that out in the image documentation? Can you use something like a Helm chart that manages these details on its own? — David Maze, Nov 30 '20 at 14:00
@DavidMaze, Using a kubernetes volume is a best practice (either persistent or ephemeral). This question here is about how handle the situation where this best practice isn't followed. — Franklin Piat, Dec 01 '20 at 08:41
Focusing on your last sentence of possible disruptions. Containers should be designed as ephemeral. A restart of a `Pod` (due to for example rescheduling) shouldn't affect the workload. If your workload is expecting the data to **not** be lost you should be using resources that Kubernetes is aware of (`PVC`'s and `PV`'s). Is there any specific reason that you need to know which `Pods` are having volumes not known by Kubernetes. What would that change? — Dawid Kruk, Dec 01 '20 at 13:42
Does this answer your question? [are VOLUME in Dockerfile persistent in kubernetes](https://stackoverflow.com/questions/65036374/are-volume-in-dockerfile-persistent-in-kubernetes) — MWZ, Dec 02 '20 at 13:21

Franklin Piat · Answer 1 · 2020-12-01T08:39:03.497

Here's a little script you can run on the Kubernetes nodes (for installation that use Docker daemon).

for d in $(docker volume ls -q --filter driver=local ) ; do
  echo "=== volume $d "
  CID=$(docker ps -q -a --filter volume=$d)
  docker container inspect  -f '  Namespace : {{ index .Config.Labels "io.kubernetes.pod.namespace"}}{{ printf "\n" }}  contnr ID : {{.ID}}{{ printf "\n" }}  Name      : {{.Name}}{{ printf "\n" }}{{ range .Mounts }}{{if eq .Driver "local" }}  Mountpoint: {{ .Destination }}  (RW:{{.RW}}){{end}}{{ end }}'  $CID
  du -ms /var/lib/docker/volumes/$d/_data | sed -e 's/^\([0-9]\+\).*/  Size      : \1 MiB/';
  echo
  echo
done

exemple:

=== volume c975149a17753393c543e25e4391af849d14d6d0cf2db4f4e873901ff05fea96
  Namespace :
  contnr ID : 01457a869b5a849952710a1bb023b10cf1f2dbf5779326d9c39a409b127a1437
  Name      : /nginx-proxy
  Mountpoint: /opt/rke-tools  (RW:true)
  Size      : 25 MiB


=== volume f8dbc7f9f06e82253ee882e19b72cbef5b30ba134230543aa34605cafcb6b082
  Namespace : foo
  contnr ID : 71758f3de8ab04b8547554eaf43e9ff54410feef597c55e628b8a3bb19c50e12
  Name      : /k8s_pgadmin4_foo-bar-001-pgadmin4-6d867f44d8-t566l_foo-bar_36715dd4-b858-4e8e-bc1c-c65cd7004e41_0
  Mountpoint: /var/lib/pgadmin  (RW:true)
  Size      : 1 MiB

Note that some system pods/containers don't have namespace.

This script should be ran on each node you want to audit.

An easy way to scan multiple host from a central server using ssh, would be to copy the script above in a file local_volumes.sh, then execute a command like cat local_volumes.sh | ssh node001 sudo bash -

For rancher users, this snippet audit all cordoned nodes: for s in $(rancher nodes ls | grep cordoned | cut -d " " -f 1 ); do cat local_volumes.sh | rancher ssh $s sudo bash - ;done

This is really quite useless in anything but a 'bedroom' installation. Once you reach more than 3 nodes it becomes a real PITA (pain in the...). Better to write a mutating admission controller that checks if all the volumes in each deployment have related PVCs and stops them if they haven't. Then you never get in this situation in the first place. Take a look at Portworx Essentials, for example. — Software Engineer, Dec 01 '20 at 08:50

List Kubernetes containers that use docker local volumes

1 Answers1