Kubernetes - MountVolume.NewMounter initialization failed for volume "" : path does not exist

Question

I am trying to setup a Local Persistent volume using local storage using WSL. But the pod STATUS stops at Pending.

The kubectl describe pod <pod-name> gives below error.

Warning FailedMount 21s (x7 over 53s) kubelet MountVolume.NewMounter initialization failed for volume "pv1" : path "/mnt/data" does not exist

The path /mnt/data has been created and exists on the local machine but cannot be accessed by the container.

and the pod and Persistent volume configuration as below.

apiVersion : v1
kind : PersistentVolume
metadata :
   name : pv1
spec :
  capacity :
    storage : 2Gi
  accessModes :
    - ReadWriteOnce
  persistentVolumeReclaimPolicy : Retain
  storageClassName : local-storage
  local : 
    fsType : ext4
    path : /mnt/data
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node1

---

apiVersion : v1
kind : PersistentVolumeClaim
metadata : 
  name : pvc1
spec :
  resources :
    requests :
      storage : 1Gi
  accessModes :
    - ReadWriteOnce
  storageClassName : local-storage

---

apiVersion : v1
kind : Pod
metadata :
  name : pod1
spec :
  containers:
  - name: www
    image: nginx:alpine
    ports:
      - containerPort: 80
        name: www
    volumeMounts:
      - name: www-store
        mountPath: /usr/share/nginx/html
  volumes :
    - name : www-store
      persistentVolumeClaim :
        claimName : pvc1

Any help would be appreciated.

Michail Alexakis · Answer 1 · 2022-05-06T23:14:14.980

9

If running on a Rancher Kubernetes Engine (RKE) cluster, this problem can arise from the fact that each kubelet also runs as a container. Thus, it does not see the filesystem of the node it runs on.

The solution is to add extra bind mounts (for kubelet service) when configuring the cluster in cluster.yml. For example, to have /data-1 on the node to be mounted as /data-1 on the kubelet:

services:
  ...
  kubelet: 
    extra_binds:
    - "/data-1:/data-1"

edited May 06 '22 at 23:14

answered Jan 25 '22 at 16:18

Michail Alexakis

1,405
15
14

Thanks for pointing this out! – perrohunter Sep 06 '22 at 21:50
Great tip, it made me think, for me I realised that microk8s had a host path storage plugin which had a default path: https://microk8s.io/docs/addon-hostpath-storage – Alex KeySmith Nov 12 '22 at 14:54

P.... · Accepted Answer · 2021-07-23T18:45:49.540

You are using nodeSelector for the pv, telling it to use node1 for the volume , chances are 1. node1 does not have /mnt/data directory present, which is hostPath for the volume. OR 2. node1 may be having /mnt/data, but the pod is getting scheduled on some other node which does not have /mnt/data directory:

apiVersion : v1
kind : PersistentVolume
metadata :
   name : pv1
spec :
  capacity :
    storage : 2Gi
  accessModes :
    - ReadWriteOnce
  persistentVolumeReclaimPolicy : Retain
  storageClassName : local-storage
  local : 
    fsType : ext4
    path : /mnt/data
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node1

Solution: Make sure /mnt/data is present on all schedulable nodes

OR

Try modifying your file to add, nodeName or nodeSelector or nodeAffinity to force it to get scheduled on the same node which have proper hostPath. In below example, it is assumed that node1 have /mnt/data directory present.

apiVersion : v1
kind : Pod
metadata :
  name : pod1
spec :
  nodeName: node1 #<------------this
  containers:
  - name: www
    image: nginx:alpine
    ports:
      - containerPort: 80
        name: www
    volumeMounts:
      - name: www-store
        mountPath: /usr/share/nginx/html
  volumes :
    - name : www-store
      persistentVolumeClaim :
        claimName : pvc1

score 0 · Answer 3 · answered Nov 19 '22 at 22:45

On Talos which similarly to the Rancher Kubernetes Engine (RKE) from the other answer also runs the kubelet containerized, one must configure extraMounts in the MachineConfig.kubelet, e.g. like this:

extraMounts:
  - destination: /var/mnt/nvme1
    type: bind
    source: /var/mnt/nvme1
    options:
      - bind
      - rshared
      - rw

shekar singh · Answer 4 · 2023-02-13T17:23:30.427

If you are using Rancher-Desktop(Windows) and facing the same issue i.e. Warning FailedMount 21s (x7 over 53s) kubelet MountVolume.NewMounter initialization failed for volume "pv1": path "/mnt/data" does not exist

As per the below document, 'For Windows, all files are automatically shared via WSL2.' https://docs.rancherdesktop.io/faq/#q-does-file-sharing-work-similarly-to-docker-desktop-do-i-have-to-do-any-additional-configuration-to-mount-volumes-to-vms

So you can create the directory inside any of the drives eg. C: or D: and during specifying the path in YAML you should prefix with /mnt

for example: suppose you created a directory inside D: drive let's say, D:\data

Now path will be like this - /mnt/d/data

Kubernetes - MountVolume.NewMounter initialization failed for volume "" : path does not exist

4 Answers4