ceph shared file system mounting error: permission denied

Question

I am facing this issue when mounting a static Ceph volume to K8s.

MountVolume.MountDevice failed for volume "test1-pv" : rpc error: code = Internal desc = an error (exit status 32) occurred while running mount args: [-t ceph  │
│ 10.107.127.65:6789,10.98.28.166:6789,10.96.128.54:6789:/volumes/sharedvg/sharedvolume/8a370586-60e6-4ec7-9d5b-c8c7ce7786c6 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/test1-pv/gl │
│ obalmount -o name=csi-cephfs-provisioner,secretfile=/tmp/csi/keys/keyfile-1586083215,mds_namespace=myfs,_netdev] stderr: mount error 13 = Permission denied

But I'm not sure if this is the right thing to do. Please point me to the right direction. Here is my use case: I want to setup a shared file system that can be accessed from all Pods in all namespaces. Concurrent write operations are not a big concern, as most of the Pods will read from this shared location, such as Python packages etc.

It is not possible by re-using the same PVC as it is a namespaced object.

What I did was to create a static volume in Ceph under a SubVolumeGroup, and create one pv-pvc pair for each namespace, and expect it will access the same files in the Ceph volume.

Here is the volume that I mounted to a Pod:

ubuntu@host1:~$ ls -l /mnt/ceph/volumes/sharedvg/sharedvolume/
total 0
drwxrwxrwx 2 root root 0 Mar  9 11:22 8a370586-60e6-4ec7-9d5b-c8c7ce7786c6
ubuntu@host1:~$ ls -l /mnt/ceph/volumes/sharedvg/sharedvolume/8a370586-60e6-4ec7-9d5b-c8c7ce7786c6/
total 0
ubuntu@host1:~$

Here is the PV and PVC yaml file. I copied the adminID and adminKey values in the secret 'rook-csi-cephfs-provisioner' of rook-ceph namespace.

apiVersion: v1
kind: Secret
metadata:
  name: rook-csi-cephfs-static-provisioner
type: Opaque
data:
  userID: "XXX"
  userKey: "XXX"

---

apiVersion: v1
kind: PersistentVolume
metadata:
  name: test1-pv
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  capacity:
    storage: 128Gi
  csi:
    driver: rook-ceph.cephfs.csi.ceph.com
    nodeStageSecretRef:
      name: rook-csi-cephfs-static-provisioner
      namespace: default
    volumeAttributes:
      clusterID: rook-ceph
      fsName: "myfs"
      staticVolume: "true"
      rootPath: /volumes/sharedvg/sharedvolume/8a370586-60e6-4ec7-9d5b-c8c7ce7786c6
    volumeHandle: test1-pv
  persistentVolumeReclaimPolicy: Retain
  volumeMode: Filesystem
  claimRef:
    name: test-pvc-1
    namespace: default

---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-pvc-1
  namespace: default
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 128Gi
  storageClassName: ""
  volumeMode: Filesystem
  volumeName: test1-pv

This is the busybox deployment yaml file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-deployment
  namespace: default
  labels:
    app: test
spec:
  replicas: 1
  strategy: 
    type: RollingUpdate
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test 
    spec:
      containers:
      - name: test 
        image: busybox
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - name: test1
          mountPath: /test1
        
        command: ['sh', '-c', 'echo Container 1 is Running ; sleep 3600']

      volumes:
      - name: test1
        persistentVolumeClaim:
          claimName: test-pvc-1

This is the log of the Pod:

   Warning  FailedMount  29m                   kubelet            MountVolume.MountDevice failed for volume "test1-pv" : rpc error: code = Internal desc = an error (exit status 32)  │
│ occurred while running mount args: [-t ceph 10.107.127.65:6789,10.98.28.166:6789,10.96.128.54:6789:/volumes/sharedvg/sharedvolume/8a370586-60e6-4ec7-9d5b-c8c7ce7786c6 /var/lib/kube │
│ let/plugins/kubernetes.io/csi/pv/test1-pv/globalmount -o name=csi-cephfs-provisioner,secretfile=/tmp/csi/keys/keyfile-973267258,mds_namespace=myfs,_netdev] stderr: mount error 13 = │
│  Permission denied                                                                                                                                                                   │
│   Warning  FailedMount  27m                   kubelet            MountVolume.MountDevice failed for volume "test1-pv" : rpc error: code = Internal desc = an error (exit status 32)  │
│ occurred while running mount args: [-t ceph 10.107.127.65:6789,10.98.28.166:6789,10.96.128.54:6789:/volumes/sharedvg/sharedvolume/8a370586-60e6-4ec7-9d5b-c8c7ce7786c6 /var/lib/kube │
│ let/plugins/kubernetes.io/csi/pv/test1-pv/globalmount -o name=csi-cephfs-provisioner,secretfile=/tmp/csi/keys/keyfile-2348945139,mds_namespace=myfs,_netdev] stderr: mount error 13  │
│ = Permission denied                                                                                                                                                                  │
│   Warning  FailedMount  25m                   kubelet            MountVolume.MountDevice failed for volume "test1-pv" : rpc error: code = Internal desc = an error (exit status 32)  │
│ occurred while running mount args: [-t ceph 10.107.127.65:6789,10.98.28.166:6789,10.96.128.54:6789:/volumes/sharedvg/sharedvolume/8a370586-60e6-4ec7-9d5b-c8c7ce7786c6 /var/lib/kube │
│ let/plugins/kubernetes.io/csi/pv/test1-pv/globalmount -o name=csi-cephfs-provisioner,secretfile=/tmp/csi/keys/keyfile-3861388178,mds_namespace=myfs,_netdev] stderr: mount error 13  │
│ = Permission denied                                                                                                                                                                  │
│   Warning  FailedMount  23m                   kubelet            MountVolume.MountDevice failed for volume "test1-pv" : rpc error: code = Internal desc = an error (exit status 32)  │
│ occurred while running mount args: [-t ceph 10.107.127.65:6789,10.98.28.166:6789,10.96.128.54:6789:/volumes/sharedvg/sharedvolume/8a370586-60e6-4ec7-9d5b-c8c7ce7786c6 /var/lib/kube │
│ let/plugins/kubernetes.io/csi/pv/test1-pv/globalmount -o name=csi-cephfs-provisioner,secretfile=/tmp/csi/keys/keyfile-4165129570,mds_namespace=myfs,_netdev] stderr: mount error 13  │
│ = Permission denied                                                                                                                                                                  │
│   Warning  FailedMount  7m14s (x10 over 34m)  kubelet            Unable to attach or mount volumes: unmounted volumes=[test1], unattached volumes=[test1 kube-api-access-fwr79]: tim │
│ ed out waiting for the condition                                                                                                                                                     │
│   Warning  FailedMount  3m3s (x13 over 21m)   kubelet            (combined from similar events): MountVolume.MountDevice failed for volume "test1-pv" : rpc error: code = Internal d │
│ esc = an error (exit status 32) occurred while running mount args: [-t ceph 10.107.127.65:6789,10.98.28.166:6789,10.96.128.54:6789:/volumes/sharedvg/sharedvolume/8a370586-60e6-4ec7 │
│ -9d5b-c8c7ce7786c6 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/test1-pv/globalmount -o name=csi-cephfs-provisioner,secretfile=/tmp/csi/keys/keyfile-2075406143,mds_namespace=myfs, │
│ _netdev] stderr: mount error 13 = Permission denied                                                                                                                                  │
│   Warning  FailedMount  27s (x2 over 29m)     kubelet            Unable to attach or mount volumes: unmounted volumes=[test1], unattached volumes=[kube-api-access-fwr79 test1]: tim │
│ ed out waiting for the condition

The Pod is in 'ContainerCreating' state.

Any suggestions? Thanks

What version of Kubernetes do you use? Is it bare-metal/cloud? — anarxz, Mar 10 '22 at 16:57
lient Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.2", GitCommit:"9d142434e3af351a628bffee3939e64c681afa4d", GitTreeState:"clean", BuildDate:"2022-01-19T17:35:46Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/arm64"} Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.2", GitCommit:"9d142434e3af351a628bffee3939e64c681afa4d", GitTreeState:"clean", BuildDate:"2022-01-19T17:29:16Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/arm64"} — ozmhsh, Mar 11 '22 at 04:30
They are VMs running in the cloud, but not the managed cluster by any cloud vendor. — ozmhsh, Mar 11 '22 at 04:31
How exactly is the cluster configured? Is it created via some distro or set up as bare-metal? — anarxz, Mar 28 '22 at 18:31

attila9778 · Answer 1 · 2023-02-23T11:03:14.697

I recently have faced the exact same issue.

Main key points in the error message which I would highlight are

MountVolume.MountDevice failed for volume ... exit status 32 ...
<your_shared_volume_path> ... mount error 13 ... Permission denied

Searching for these these essential keywords and error codes led me to posts that explained the 'Permission denied' error.

Firstly, focusing only on the mount command's error I have found this post's first comment's highlight:

Permissions on credentials file

Unfortunately, in this case further logs cannot be retrived while the pod's shell in 'ContainerCreating' state is not accessible.

Secondly, searching for both error codes at once led me to the solution mentioning

observed secret for fileshare was not correct

Having in my mind that credential changes happened not so late, put together to check my Kubernetes secrets where these credentials are stored.

Updating these storage access credentials in the Kubernetes secret solved the issue, enabling the pod to enter 'Running' state after the next restart.

ceph shared file system mounting error: permission denied

1 Answers1