I am deploying Airflow with official Helm chart and trying to understand why it requires stateful set for worker deployment. When it makes perfect sense for redis and postgtre I am not sure why this is requirement for worker
2 Answers
Actually, in the official Helm chart, they choose between statefulset
and deployment
to deploy your workers, based on your persistence configurations:
- if it's enabled (by default), they use the
StatefulSet
in order to create the PVC which will create a PV per pod. - and if it's disabled, they use the
deployment
Here is the link to the condition they use to choose between the two ressources.

- 4,285
- 2
- 9
- 23
-
Actually I have deployed with persistance disabled: persistence: # Enable persistent volumes enabled: false but it's still deploying worker as statefulset kubectl get statefulset -n dw NAME READY AGE airflow-postgresql 1/1 6d3h airflow-redis 1/1 6d3h airflow-worker 2/2 6d3h – Paweł Feliks Aug 18 '22 at 12:05
-
Check if you are using the last version of the helm chart (1.6.0), and if you set `workers.persistence.enabled` to false and no other value, I'm using this version without persistence, and the workers are deployed as deployment – Hussein Awala Aug 18 '22 at 12:58
-
Thank you for your help - actually you were 100% correct I upgraded value but in copy file. After chaning flag enabled to false in persistentce section it's now stateless. – Paweł Feliks Aug 22 '22 at 13:28
At least originally, the workers were deployed as a StatefulSet because the logs for the jobs were stored on the persistent volume associated with the id. When the webserver requests the logs from the worker, it needed to reference the defacto id in the set, e.g. celery-0, celery-1. If the webserver queried a random worker, it would return no logs, and was a common "bug/problem" people experience during deployment.
This may or may not be true today, but it is clearly explained here: https://artifacthub.io/packages/helm/airflow-helm/airflow/7.15.0#docs-kubernetes---worker-autoscaling

- 103
- 5
-
Of course, I'd appreciate accepting my answer as the correct answer, since it actually answers the original question. – Joe Sep 11 '22 at 17:14