How to bind a PVC with worker pod in airflow when using kubernetes executor?

Question

I am using Kubernetes executor in airflow and for data management (sharing data across pod), trying to write data in some mounted directory but the issue that I am facing is how to mount PVC over the worker pod where that task executes.

Worker pod spawns only at the time of dag execution and get deleted after completion of the task. Since the worker pod is only manageable through the configuration file (airflow.cfg).

How can bind a PVC to worker pod other than used for dags and logs?

Have you already solved problem ? – Malgorzata Dec 30 '19 at 13:29 — Malgorzata, Dec 30 '19 at 13:29

score 0 · Answer 1 · answered Dec 11 '19 at 10:13

When you delete a PVC, corresponding PV becomes Released. This PV can contain sensitive data (say credit card numbers) and therefore nobody can ever bind to it, even if it is a PVC with the same name and in the same namespace as the previous one - who knows who's trying to steal the data!

Action is required here. You have two options:

Make the PV available to everybody - delete PV.Spec.ClaimRef, Such PV can bound to any PVC (assuming that capacity, access mode and selectors match)
Make the PV available to a specific PVC - pre-fill PV.Spec.ClaimRef with a pointer to a PVC. Leave the PV.Spec.ClaimRef,UID empty, as the PVC does not to need exist at this point and you don't know PVC's UID. This PV can be bound only to the specified PVC.

Simply create new PVC bound to exiting PV (via its UID) and deployment that uses this PV.

How to bind a PVC with worker pod in airflow when using kubernetes executor?

1 Answers1