Kubernetes volumes not getting mounted

Question

I'm deploying a device plugin for FPGAs on a local kubernetes cluster. Essentially it is just a daemon set, so each node in the cluster (barring master nodes) will have one pod of this deployment.

The pods need to access the device trees of the hosts (nodes), they also need to access the kubelet socket. So I mount two specific directories from the hosts to the containers, as follows:

      containers:
      - image: uofthprc/fpga-k8s-deviceplugin
        name: fpga-device-plugin-ctr
        volumeMounts:
          - name: device-plugin
            mountPath: /var/lib/kubelet/device-plugins
          - name: device-info
            mountPath: /sys/firmware/devicetree/base
            readOnly: true
      volumes:
        - name: device-plugin
          hostPath:
            path: /var/lib/kubelet/device-plugins
        - name: device-info
          hostPath:
            path: /sys/firmware/devicetree/base

For some reason, the /var/lib/kubelet/device-plugins mounts fine, and is fully accessible from within the containers, while the /sys/firmware/devicetree/base is not!. The following is the output of attaching to one of the containers kubectl exec -it fpga-device-plugin-ds-hr6s5 -n device-plugins -- /bin/sh:

/work # ls /var/lib/kubelet/device-plugins
DEPRECATION                  kubelet.sock                 kubelet_internal_checkpoint
/work # ls /sys/firmware/devicetree/base
ls: /sys/firmware/devicetree/base: No such file or directory
/work # ls /sys/firmware/devicetree
ls: /sys/firmware/devicetree: No such file or directory
/work # ls /sys/firmware
/work # ls /sys
block     bus       class     dev       devices   firmware  fs        kernel    module    power
/work #

I'm not sure why this happens, but I tested this with Read Only permissions, Read Write permissions, and without the mount at all. In all three cases, there's nothing inside the path /sys/firmware in the containers. On the hosts, I'm 100% sure that the path /sys/firmware/devicetree/base/ exists and includes the files I want.

Here's the output of describe pods on one of the containers:

Name:                 fpga-device-plugin-ds-hr6s5
Namespace:            device-plugins
Priority:             2000001000
Priority Class Name:  system-node-critical
Node:                 mpsoc2/10.84.31.12
Start Time:           Wed, 20 May 2020 22:56:25 -0400
Labels:               controller-revision-hash=cfbc8976f
                      name=fpga-device-plugin-ds
                      pod-template-generation=1
Annotations:          cni.projectcalico.org/podIP: 10.84.32.223/32
                      cni.projectcalico.org/podIPs: 10.84.32.223/32
Status:               Running
IP:                   10.84.32.223
IPs:
  IP:           10.84.32.223
Controlled By:  DaemonSet/fpga-device-plugin-ds
Containers:
  fpga-device-plugin-ctr:
    Container ID:   docker://629ab2fd7d05bc17e6f566912b127eec421f214123309c10674c40ed2839d1cf
    Image:          uofthprc/fpga-k8s-deviceplugin
    Image ID:       docker-pullable://uofthprc/fpga-k8s-deviceplugin@sha256:06f9e46470219d5cfb2e6233b1473e9f1a2d3b76c9fd2d7866f7a18685b60ea3
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Wed, 20 May 2020 22:56:29 -0400
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /sys/firmware/devicetree/base from device-info (ro)
      /var/lib/kubelet/device-plugins from device-plugin (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-dwbsm (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  device-plugin:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/device-plugins
    HostPathType:  
  device-info:
    Type:          HostPath (bare host directory volume)
    Path:          /sys/firmware/devicetree/base
    HostPathType:  
  default-token-dwbsm:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-dwbsm
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/pid-pressure:NoSchedule
                 node.kubernetes.io/unreachable:NoExecute
                 node.kubernetes.io/unschedulable:NoSchedule
Events:
  Type    Reason     Age        From               Message
  ----    ------     ----       ----               -------
  Normal  Scheduled  <unknown>  default-scheduler  Successfully assigned device-plugins/fpga-device-plugin-ds-hr6s5 to mpsoc2
  Normal  Pulling    23s        kubelet, mpsoc2    Pulling image "uofthprc/fpga-k8s-deviceplugin"
  Normal  Pulled     23s        kubelet, mpsoc2    Successfully pulled image "uofthprc/fpga-k8s-deviceplugin"
  Normal  Created    23s        kubelet, mpsoc2    Created container fpga-device-plugin-ctr
  Normal  Started    22s        kubelet, mpsoc2    Started container fpga-device-plugin-ctr

As far as I can see, no problems in it.

I'm using kubernetes (kubeadm installed, not microk8s) version 1.18.2 for the client and server. The nodes at question are ARM64 nodes with Ubuntu 16.04, using a 4.14.0 kernel. The containers are all alpine:3.11 with a simple binary copied inside them. I have no idea why the mount is not working, any help would certainly be appreciated.

Edit1:

The permissions of /sys/firmware/devicetree/base/ on the hosts are as follows:

savi@mpsoc10:~$ ls -alh /sys/firmware/devicetree/base/
total 0
drwxr-xr-x 36 root root  0 May 20 21:32 .
drwxr-xr-x  3 root root  0 May 20 21:32 ..
-r--r--r--  1 root root  4 May 20 21:32 #address-cells
drwxr-xr-x  2 root root  0 May 20 21:32 aliases
drwxr-xr-x 56 root root  0 May 20 21:32 amba
drwxr-xr-x  3 root root  0 May 20 21:32 amba_apu@0
drwxr-xr-x  2 root root  0 May 20 21:32 aux_ref_clk
-r--r--r--  1 root root 15 May 20 21:32 board
drwxr-xr-x  2 root root  0 May 20 21:32 chosen
drwxr-xr-x  2 root root  0 May 20 21:32 clk
-r--r--r--  1 root root 12 May 20 21:32 compatible
drwxr-xr-x  6 root root  0 May 20 21:32 cpu_opp_table
drwxr-xr-x  7 root root  0 May 20 21:32 cpus
drwxr-xr-x  2 root root  0 May 20 21:32 dcc
drwxr-xr-x  2 root root  0 May 20 21:32 dp_aclk
drwxr-xr-x  2 root root  0 May 20 21:32 edac
drwxr-xr-x  2 root root  0 May 20 21:32 fclk0
drwxr-xr-x  2 root root  0 May 20 21:32 fclk1
drwxr-xr-x  2 root root  0 May 20 21:32 fclk2
drwxr-xr-x  2 root root  0 May 20 21:32 fclk3
drwxr-xr-x  3 root root  0 May 20 21:32 firmware
drwxr-xr-x  2 root root  0 May 20 21:32 fpga-full
drwxr-xr-x  2 root root  0 May 20 21:32 gt_crx_ref_clk
drwxr-xr-x  2 root root  0 May 20 21:32 mailbox@ff990400
drwxr-xr-x  2 root root  0 May 20 21:32 memory
-r--r--r--  1 root root  1 May 20 21:32 name
drwxr-xr-x  3 root root  0 May 20 21:32 nvmem_firmware
drwxr-xr-x  2 root root  0 May 20 21:32 pcap
drwxr-xr-x  2 root root  0 May 20 21:32 pmu
drwxr-xr-x 31 root root  0 May 20 21:32 power-domains
drwxr-xr-x  2 root root  0 May 20 21:32 psci
drwxr-xr-x  2 root root  0 May 20 21:32 pss_alt_ref_clk
drwxr-xr-x  2 root root  0 May 20 21:32 pss_ref_clk
drwxr-xr-x  2 root root  0 May 20 21:32 reset-controller
drwxr-xr-x  2 root root  0 May 20 21:32 sha384
-r--r--r--  1 root root  4 May 20 21:32 #size-cells
drwxr-xr-x  2 root root  0 May 20 21:32 __symbols__
drwxr-xr-x  2 root root  0 May 20 21:32 timer
-r--r--r--  1 root root 10 May 20 21:32 vendor
drwxr-xr-x  2 root root  0 May 20 21:32 video_clk
drwxr-xr-x  2 root root  0 May 20 21:32 zynqmp-power
drwxr-xr-x  2 root root  0 May 20 21:32 zynqmp_rsa

some of the files inside it are read only, which is what prompted me to use read only permissions for the volume mount in the first place.

The following is the permissions of /sys and /sys/firmware on the containers:

/work # ls -alh /sys/
total 4K     
dr-xr-xr-x   12 root     root           0 May 21 02:56 .
drwxr-xr-x    1 root     root        4.0K May 21 02:56 ..
drwxr-xr-x    2 root     root           0 May 21 03:08 block
drwxr-xr-x   32 root     root           0 May 21 03:08 bus
drwxr-xr-x   64 root     root           0 May 21 03:08 class
drwxr-xr-x    4 root     root           0 May 21 03:08 dev
drwxr-xr-x    9 root     root           0 May 21 03:08 devices
drwxrwxrwt    2 root     root          40 May 21 02:56 firmware
drwxr-xr-x   10 root     root           0 May 21 02:56 fs
drwxr-xr-x    7 root     root           0 May 21 02:56 kernel
drwxr-xr-x  156 root     root           0 May 21 03:08 module
drwxr-xr-x    2 root     root           0 May 21 03:08 power
/work # ls -alh /sys/firmware/
total 0      
drwxrwxrwt    2 root     root          40 May 21 02:56 .
dr-xr-xr-x   12 root     root           0 May 21 02:56 ..

EDIT2:

The output of mount | grep sysfs on containers is:

/work # mount | grep sysfs
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
sysfs on /sys/firmware/devicetree/base type sysfs (ro,relatime)

please add the permission of the device tree folder (on the host) — omricoco, May 21 '20 at 11:12
It might caused by the fact that sysfs is mounted as readonly. When you run `mount | grep sysfs` inside container what is the output?https://github.com/moby/moby/issues/24000 — kool, May 21 '20 at 14:35
@KFC_ you may be onto something here, it is mounted as read only! I'm reading through the github issue trying to see if there's a fix. — mewais, May 21 '20 at 14:52

kool · Accepted Answer · 2020-05-21T16:21:43.260

sysfs is mounted as read-only:

mount | grep sysfs
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)

And that's why the volume is not shown in the pod. You can change it to writable volume by using init container and running both containers as privileged. If ran without privileged: true the access will not be modified and volume won't be mounted to a pod:

initContainers:
  - name: mount
    image: nginx:alpine
    command: ["/bin/sh", "-c", "mount -o remount,rw '/sys'"]
    securityContext:
      privileged: true

This way it will change /sys to writable:

mount | grep sysfs
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys/firmware/ type sysfs (rw,relatime)

Kubernetes volumes not getting mounted

Edit1:

EDIT2:

1 Answers1

Linked