I want to use Airflow in Kubernetes on my local machine. From the Airflow helm chart doc I should use a PVC to use my local DAG files, so I setup my PV and PVC like so:
apiVersion: v1
kind: PersistentVolume
metadata:
name: dags-pv
spec:
volumeMode: Filesystem
storageClassName: local-path
capacity:
storage: 10Gi
accessModes:
- ReadWriteMany
hostPath:
path: /mnt/c/Users/me/dags
type: Directory
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: dags-pvc
spec:
storageClassName: local-path
accessModes:
- ReadWriteMany
resources:
requests:
storage: 3Gi
Then I create a override-values.yaml file
config:
core:
dags_folder: "/usr/somepath/airflow/dags"
dags:
persistence:
enabled: true
existingClaim: "dags-pvc"
gitSync:
enabled: false
Note here that I want to change the default DAG folder path. And that's where I am having difficulties (it works if I keep the default DAG folder path). I don't know how to create a mount point and attach the PVC to it.
I tried to add, in my override file:
worker:
extraVolumeMounts:
- name: w-dags
mountPath: "/usr/somepath/airflow/dags"
extraVolumes:
- name: w-dags
persistentVolumeClaim:
claimName: "dags-pvc"
scheduler:
extraVolumeMounts:
- name: s-dags
mountPath: "/usr/somepath/airflow/dags"
extraVolumes:
- name: s-dags
persistentVolumeClaim:
claimName: "dags-pvc"
But that doesn't work, my scheduler is stuck on Init:0/1: "Unable to attach or mount volumes: unmounted volumes=[dags], unattached volumes=[logs dags s-dags config kube-api-access-9mc4c]: timed out waiting for the condition". So, I can tell I broke a condition - dags should be mounted (aka my extraVolumes section is wrong) - but I am not sure where to go from here.
