Can I "recycle" Pods when using Kubernetes Executor on Airflow, to avoid repeating the Initialization?

Question

Context: Running Airflow v2.2.2 with Kubernetes Executor.

I am running a process that creates a burst of quick tasks.

The tasks are short enough that the Kubernetes Pod initialization takes up the majority of runtime for many of them.

Is there a way to re-utilize pre-initialized pods for multiple tasks?

I've found a comment in an old issue that states that when running the Subdag operator, all subtasks will be run on the same pod, but I cannot find any further information. Is this true?

I have searched the following resources:

• Airflow Documentation: Kubernetes
• Airflow Documentation: KubernetesExecutor
• Airflow Documentation: KubernetesPodOperator
• StackOverflow threads: Run two jobs on same pod, Best Run Airflow on Kube
• Google Search: airflow kubernetes reuse initialization

But haven't really found anything that directly addresses my problem.

Answer 1

I don't think this is possible in Airlfow even with Subdag operator which runs a separate dag as a part of the current dag with the same way used for the other tasks.

To solve your problem, you can use CeleryKubernetesExecutor which combine CeleryExecutor and KubernetesExecutor. By default the tasks will be queued in Celery queue but for the heavy tasks, you can choose K8S queue in order to run them in isolated pods. In this case you will be able to use the Celery workers which are up all the time.

kubernetes_task= BashOperator(
        task_id='kubernetes_executor_task',
        bash_command='echo "Hello from Kubernetes"',
        queue = 'kubernetes'
    )
celery_task = BashOperator(
        task_id='celery_executor_task',
        bash_command='echo "Hello from Celery"',
    )

If you are worry about the scalability/cost of Celery workers, you can use KEDA to scale the Celery workers from 0 workers to a maximum number of workers based on queued tasks count.