0

In my Kubernetes pods with mounted volumes, I don't seem to have access to the underlying disk device in the /dev folder. I need that for the XFS tools to work. I am running the cluster on DigitalOcean.

The example volume is mounted on /var/www. The output of df from a running pod is:

$ df -h 
Filesystem                                                                Size  Used Avail Use% Mounted on
overlay                                                                   158G  9.7G  142G   7% /
tmpfs                                                                      64M     0   64M   0% /dev
tmpfs                                                                     3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/disk/by-id/scsi-0DO_Volume_pvc-650ccba6-3177-45b5-9ffb-0ac2a931fddc  1.0M  1.0M     0 100% /var/www
/dev/vda1                                                                 158G  9.7G  142G   7% /etc/hosts
shm                                                                        64M     0   64M   0% /dev/shm
tmpfs                                                                     3.9G   12K  3.9G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                                                                     3.9G     0  3.9G   0% /proc/acpi
tmpfs                                                                     3.9G     0  3.9G   0% /sys/firmware

However, the output of lsblk does not reveal any such device /dev/disk/by-id/scsi-0DO_Volume_pvc-650ccba6-3177-45b5-9ffb-0ac2a931fddc; it shows a /dev/sdb device instead.

$ lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0    1G  0 disk 
sdb      8:16   0    1G  0 disk /var/www
sdc      8:32   0    1G  0 disk 
sde      8:64   0    1G  0 disk 
vda    254:0    0  160G  0 disk 
|-vda1 254:1    0  160G  0 part /etc/apache2/sites-available
`-vda2 254:2    0    2M  0 part 
vdb    254:16   0  472K  1 disk

None of the devices are avaiable in my pod:

$ ls -la /dev
total 4
drwxr-xr-x 5 root root  360 Oct 27 10:59 .
drwxr-xr-x 1 root root 4096 Oct 27 10:59 ..
lrwxrwxrwx 1 root root   11 Oct 27 10:59 core -> /proc/kcore
lrwxrwxrwx 1 root root   13 Oct 27 10:59 fd -> /proc/self/fd
crw-rw-rw- 1 root root 1, 7 Oct 27 10:59 full
drwxrwxrwt 2 root root   40 Oct 27 10:58 mqueue
crw-rw-rw- 1 root root 1, 3 Oct 27 10:59 null
lrwxrwxrwx 1 root root    8 Oct 27 10:59 ptmx -> pts/ptmx
drwxr-xr-x 2 root root    0 Oct 27 10:59 pts
crw-rw-rw- 1 root root 1, 8 Oct 27 10:59 random
drwxrwxrwt 2 root root   40 Oct 27 10:58 shm
lrwxrwxrwx 1 root root   15 Oct 27 10:59 stderr -> /proc/self/fd/2
lrwxrwxrwx 1 root root   15 Oct 27 10:59 stdin -> /proc/self/fd/0
lrwxrwxrwx 1 root root   15 Oct 27 10:59 stdout -> /proc/self/fd/1
-rw-rw-rw- 1 root root    0 Oct 27 10:59 termination-log
crw-rw-rw- 1 root root 5, 0 Oct 27 10:59 tty
crw-rw-rw- 1 root root 1, 9 Oct 27 10:59 urandom
crw-rw-rw- 1 root root 1, 5 Oct 27 10:59 zero

I also tried to change runAsUser, runAsGroup and added Linux capabilities such as SYS_ADMIN and SYS_RESOURCE - it didn't help.

When I set the container to run as privileged, I had access to the /dev/sdb and then I could symlink it to /dev/disk/by-id/scsi-0DO_Volume_pvc-650ccba6-3177-45b5-9ffb-0ac2a931fddc so that the XFS tools can run.

However, it feels like a hack to create the symlink manually and also it only works with privileged containers:

SOURCE_LINK=$(df -h | grep -- "/var/www" | cut -d" " -f1)
TARGET_DEVICE=$(lsblk | grep -- "/var/www" | cut -d" " -f1)
mkdir -p $(dirname $SOURCE_LINK)
ln -s "/dev/$TARGET_DEVICE" "$SOURCE_LINK"

So my question is: is there some kind of configuration / settings / approach that would reveal the /dev/disk/by-id/scsi-0DO_Volume_pvc-650ccba6-3177-45b5-9ffb-0ac2a931fddc in the pod, ideally in the non-privileged mode?

(This is related to another question of mine on how to enable the XFS quotas.)

Aleš Krajník
  • 177
  • 1
  • 10
  • 1
    Generally you run things in containers specifically so they can't directly access the host system hardware (and, especially, raw disk block devices that would let you bypass all system controls). This task is probably better run directly on the Kubernetes nodes outside of containers. – David Maze Oct 27 '20 at 17:30
  • Thank you @DavidMaze! However, I don't have access to the node and also need to run this as a part of a service that changes the quotas dynamically. Having said that, Kubernetes might not be the best platform for these. – Aleš Krajník Nov 03 '20 at 17:02

1 Answers1

1

This question was answered in another question of mine by Timo from DigitalOcean.

In order to get access to the device, you also need to mount the host's /dev directory to the container.

volumes:
  - name: device-dir
    hostPath:
      path: /dev

and for the container:

volumeMounts:
  - name: device-dir
    mountPath: /dev

This removes the need to create the symlink manually as it already exists in the host. However, the device won't be writable. As explained in the Kubernetes documentation, the container needs to run as privileged and root in order to be able to write to such directories or the host's directories needs to have their permissions adjusted.

Aleš Krajník
  • 177
  • 1
  • 10