3

I'm trying to set the request_cpu parameter in the Kubernetes executor for Airflow but haven't been able to find where I can do that. In the default airflow config I found default_cpus but according to this answer there is nowhere that that is used, and nowhere else in the Kubernetes section could I find a reference to the CPU request.

How can I set the request_cpu parameter in the Airflow Kubernetes executor?

EDIT: Ideally, what I would like to be able to do is set this as a global default rather than on a per-operator basis, even though in general I think that it does make more sense to set it per-operator/task

qfwfq
  • 2,416
  • 1
  • 17
  • 30

1 Answers1

4

You can set this executor_config at task level for KubernetesExecutor as follows. 

    exmaple_task = PythonOperator(
        task_id="exmaple_task",
        python_callable=print_stuff,
        executor_config={
            "KubernetesExecutor": {"request_cpu": "1",
                                   "request_memory": "128Mi",
                                   "limit_memory": "128Mi"}}
    )

You can define the following in executor_config :

  • image
  • image_pull_policy
  • request_memory
  • request_cpu
  • limit_memory
  • limit_cpu
  • limit_gpu
  • node_selectors
  • affinity
  • tolerations
  • annotations
  • volumes
  • volume_mounts
  • volume_mounts

Doc: https://airflow.apache.org/docs/1.10.9/_api/airflow/contrib/executors/kubernetes_executor/index.html#airflow.contrib.executors.kubernetes_executor.KubernetesExecutorConfig

kaxil
  • 17,706
  • 2
  • 59
  • 78
  • Can a default be set globally, though? – qfwfq Feb 27 '20 at 23:36
  • 1
    You can set it in default_args for the DAG, so you don't need to duplicate per task. But to set it globally you can enforce policy using pod_mutation_hook (https://airflow.apache.org/docs/stable/kubernetes.html#pod-mutation-hook) – kaxil Feb 27 '20 at 23:41
  • Hmm. It's unfortunate that there's not an easier way to do that. Thanks for the point to the pod_mutation_hook, I'll check that out – qfwfq Feb 27 '20 at 23:48
  • @kaxil, Where should the pod_mutation_hook function defined. Is it defined for each DAG file? I have tried to add a test annotation for KubernetesExecutor, but it doesn't seem to work. – fuyi Mar 21 '20 at 22:58
  • The pod_mutation_hook should be defined in `airflow_local_settings.py` file and this file should be on the PYTHONPATH. So it can stay in `$AIRFLOW_HOME/configs/airflow_local_settings.py` – kaxil Mar 22 '20 at 01:59
  • Hi @kaxil, I have tried to create a pod_mutation_hook function in airflow_local_settings.py and put it under $AIRFLOW_HOME/configs/airflow_local_settings.py, wich which set the annotation pod.annotations['airflow.apache.org/launched-by'] = 'Tests'. However, this annotation is not picked up by KubernetesExecututor Pod. I have no idea, Do you have a good example code of pod_mutation_hook? – fuyi Mar 23 '20 at 17:13
  • @fuyi here is an example: https://github.com/astronomer/airflow-chart/blob/master/templates/configmap.yaml#L131-L152 – kaxil Mar 23 '20 at 19:27
  • I tried this on airflow v2.4.1 but got `[2022-11-23, 11:18:36 UTC] {logging_mixin.py:117} WARNING - /home/airflow/.local/lib/python3.7/site-packages/airflow/kubernetes/pod_generator.py:176 RemovedInAirflow3Warning: Using a dictionary for the executor_config is deprecated and will soon be removed.please use a `kubernetes.client.models.V1Pod` class with a "pod_override" key instead.` Does anyone know an example of the new approach it wants? – andrewm4894 Nov 23 '22 at 11:25