Unable to attach or mount volumes: unmounted volumes

Question

i deplyed my application on kubernetes but have been getting this error:

**MountVolume.SetUp failed for volume "airflow-volume" : mount failed: mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/4a3c3d0b-b7e8-49bc-8a78-5a8bdc932eca/volumes/kubernetes.io~glusterfs/airflow-volume --scope -- mount -t glusterfs -o auto_unmount,backup-volfile-servers=10.0.2.107:10.0.2.24,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/airflow-volume/worker-844c9db787-vprt8-glusterfs.log,log-level=ERROR 10.0.2.107:/airflow /var/lib/kubelet/pods/4a3c3d0b-b7e8-49bc-8a78-5a8bdc932eca/volumes/kubernetes.io~glusterfs/airflow-volume Output: Running scope as unit run-22059.scope. mount: /var/lib/kubelet/pods/4a3c3d0b-b7e8-49bc-8a78-5a8bdc932eca/volumes/kubernetes.io~glusterfs/airflow-volume: unknown filesystem type 'glusterfs'. , the following error information was pulled from the glusterfs log to help diagnose this issue: could not open log file for pod worker-844c9db787-vprt8**

AND

**Unable to attach or mount volumes: unmounted volumes=[airflow-volume], unattached volumes=[airflow-volume default-token-s6pvd]: timed out waiting for the condition**

Any suggestions?

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: airflow
spec:
  replicas: 1
  selector:
    matchLabels:
      tier: web
  template:
    metadata:
      labels:
        app: airflow
        tier: web
    spec:
      imagePullSecrets:
        - name: peeriqregistrykey
      restartPolicy: Always
      containers:
        # Airflow Webserver Container
        - name: web
          image: peeriq/data_availability_service:airflow-metadata-cutover
          volumeMounts:
            - mountPath: /usr/local/airflow
              name: airflow-volume
          envFrom:
            - configMapRef:
                name: airflow-config
          env:
            - name: VAULT_ADDR
              valueFrom:
                secretKeyRef:
                  name: vault-credentials
                  key: VAULT_ADDR
            - name: VAULT_TOKEN
              valueFrom:
                secretKeyRef:
                  name: vault-credentials
                  key: VAULT_TOKEN
            - name: DJANGO_AUTH_USER
              valueFrom:
                secretKeyRef:
                  name: django-auth
                  key: DJANGO_AUTH_USER
            - name: DJANGO_AUTH_PASS
              valueFrom:
                secretKeyRef:
                  name: django-auth
                  key: DJANGO_AUTH_PASS
            - name: FERNET_KEY
              valueFrom:
                secretKeyRef:
                  name: airflow-secrets
                  key: FERNET_KEY
            - name: POSTGRES_SERVICE_HOST
              valueFrom:
                secretKeyRef:
                  name: rds-postgres
                  key: POSTGRES_SERVICE_HOST
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: rds-postgres
                  key: POSTGRES_PASSWORD
          ports:
            - name: web
              containerPort: 8080
          args: ["webserver"]
        # Airflow Scheduler Container
        - name: scheduler
          image: peeriq/data_availability_service:airflow-metadata-cutover
          volumeMounts:
            - mountPath: /usr/local/airflow
              name: airflow-volume
          envFrom:
            - configMapRef:
                name: airflow-config
          env:
            - name: AWS_DEFAULT_REGION
              value: us-east-1
            - name: ETL_AWS_ACCOUNT_NUMBER
              valueFrom:
                secretKeyRef:
                  name: aws-creds
                  key: ETL_AWS_ACCOUNT_NUMBER
            - name: VAULT_ADDR
              valueFrom:
                secretKeyRef:
                  name: vault-credentials
                  key: VAULT_ADDR
            - name: VAULT_TOKEN
              valueFrom:
                secretKeyRef:
                  name: vault-credentials
                  key: VAULT_TOKEN
            - name: DJANGO_AUTH_USER
              valueFrom:
                secretKeyRef:
                  name: django-auth
                  key: DJANGO_AUTH_USER
            - name: DJANGO_AUTH_PASS
              valueFrom:
                secretKeyRef:
                  name: django-auth
                  key: DJANGO_AUTH_PASS
            - name: FERNET_KEY
              valueFrom:
                secretKeyRef:
                  name: airflow-secrets
                  key: FERNET_KEY
            - name: POSTGRES_SERVICE_HOST
              valueFrom:
                secretKeyRef:
                  name: rds-postgres
                  key: POSTGRES_SERVICE_HOST
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: rds-postgres
                  key: POSTGRES_PASSWORD
          args: ["scheduler"]
      volumes:
        - name: airflow-volume
          # This GlusterFS volume must already exist.
          glusterfs:
            endpoints: glusterfs-cluster
            path: /airflow
            readOnly: false
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: flower
  namespace: airflow
spec:
  replicas: 1
  selector:
    matchLabels:
      tier: flower
  template:
    metadata:
      labels:
        app: airflow
        tier: flower
    spec:
      imagePullSecrets:
        - name: peeriqregistrykey
      restartPolicy: Always
      containers:
        - name: flower
          image: peeriq/data_availability_service:airflow-metadata-cutover
          volumeMounts:
            - mountPath: /usr/local/airflow
              name: airflow-volume
          envFrom:
            - configMapRef:
                name: airflow-config
          env:
            # To prevent the error: ValueError: invalid literal for int() with base 10: 'tcp://10.0.0.83:5555'
            - name: FLOWER_PORT
              value: "5555"
            - name: DJANGO_AUTH_USER
              valueFrom:
                secretKeyRef:
                  name: django-auth
                  key: DJANGO_AUTH_USER
            - name: DJANGO_AUTH_PASS
              valueFrom:
                secretKeyRef:
                  name: django-auth
                  key: DJANGO_AUTH_PASS
            - name: POSTGRES_SERVICE_HOST
              valueFrom:
                secretKeyRef:
                  name: rds-postgres
                  key: POSTGRES_SERVICE_HOST
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: rds-postgres
                  key: POSTGRES_PASSWORD
          ports:
            - name: flower
              containerPort: 5555
          args: ["flower"]
      volumes:
        - name: airflow-volume
          # This GlusterFS volume must already exist.
          glusterfs:
            endpoints: glusterfs-cluster
            path: /airflow
            readOnly: false
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: worker
  namespace: airflow
spec:
  replicas: 1
  selector:
    matchLabels:
      tier: worker
  template:
    metadata:
      labels:
        app: airflow
        tier: worker
    spec:
      imagePullSecrets:
        - name: peeriqregistrykey
      restartPolicy: Always
      containers:
        - name: worker
          image: peeriq/data_availability_service:airflow-metadata-cutover
          volumeMounts:
            - mountPath: /usr/local/airflow
              name: airflow-volume
          envFrom:
            - configMapRef:
                name: airflow-config
          env:
            - name: AWS_DEFAULT_REGION
              value: us-east-1
            - name: ETL_AWS_ACCOUNT_NUMBER
              valueFrom:
                secretKeyRef:
                  name: aws-creds
                  key: ETL_AWS_ACCOUNT_NUMBER
            - name: VAULT_ADDR
              valueFrom:
                secretKeyRef:
                  name: vault-credentials
                  key: VAULT_ADDR
            - name: VAULT_TOKEN
              valueFrom:
                secretKeyRef:
                  name: vault-credentials
                  key: VAULT_TOKEN
            - name: DJANGO_AUTH_USER
              valueFrom:
                secretKeyRef:
                  name: django-auth
                  key: DJANGO_AUTH_USER
            - name: DJANGO_AUTH_PASS
              valueFrom:
                secretKeyRef:
                  name: django-auth
                  key: DJANGO_AUTH_PASS
            - name: FERNET_KEY
              valueFrom:
                secretKeyRef:
                  name: airflow-secrets
                  key: FERNET_KEY
            - name: POSTGRES_SERVICE_HOST
              valueFrom:
                secretKeyRef:
                  name: rds-postgres
                  key: POSTGRES_SERVICE_HOST
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: rds-postgres
                  key: POSTGRES_PASSWORD
          args: ["worker"]
      volumes:
        - name: airflow-volume
          # This GlusterFS volume must already exist.
          glusterfs:
            endpoints: glusterfs-cluster
            path: /airflow
            readOnly: false

Have you installed GlusterFS on your cluster? Check this: https://kubernetes.io/docs/concepts/storage/volumes/#glusterfs — Alif Biswas, Jan 27 '21 at 06:48

score 3 · Accepted Answer · answered Jan 27 '21 at 07:20

3

You must install package glusterfs-fuse on your Kubernetes nodes, otherwise it won't be able to mount glusterfs volumes.

answered Jan 27 '21 at 07:20

Vasili Angapov

8,061
15
31

Hey @vasili is glusterfs-fuse for ubuntu? and if not, is there anything recommended for ubuntu? – Gaurav Jaitly Jan 29 '21 at 22:59
Read more here: https://docs.gluster.org/en/latest/Administrator-Guide/Setting-Up-Clients/ – Vasili Angapov Jan 30 '21 at 06:58

score 0 · Answer 2 · answered Jan 27 '21 at 01:08

0

The part of the message unknown filesystem type 'glusterfs' can mean that there is something wrong with your volume definition or a storage class if you use it. But this is a guess.

answered Jan 27 '21 at 01:08

Alex

81
2

but this has been working on prod env. i tried it on test env. the difference is test env k8s is on EKS. this shouldnt make difference but this has been working for prod. – Gaurav Jaitly Jan 27 '21 at 05:31

score 0 · Answer 3 · edited Nov 10 '21 at 07:53

0

I had the same error, and the reason in my kubernetes was that the nfs server was unavailable. After starting the nfs server, it was solved.

edited Nov 10 '21 at 07:53

ouflak

2,458
10
44
49

answered Nov 10 '21 at 06:46

Alex

1

This does not really answer the question. If you have a different question, you can ask it by clicking [Ask Question](https://stackoverflow.com/questions/ask). To get notified when this question gets new answers, you can [follow this question](https://meta.stackexchange.com/q/345661). Once you have enough [reputation](https://stackoverflow.com/help/whats-reputation), you can also [add a bounty](https://stackoverflow.com/help/privileges/set-bounties) to draw more attention to this question. - [From Review](/review/late-answers/30303045) – AfterFray Nov 11 '21 at 05:19

Unable to attach or mount volumes: unmounted volumes

3 Answers3