3

Airflow - Unable to use jinja template for resources in Kubernetes Pod Operator task. Able to use jinja template for environment variables, image but not able to use for resources to specify CPU and memory (request and limit).

Raja Sekhar
  • 31
  • 1

1 Answers1

1

This happens because resources is not a templated field (see source code). You can still get the functionality you desire by creating a custom operator as:

from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator
class MyKubernetesPodOperator(KubernetesPodOperator):
    template_fields = KubernetesPodOperator.template_fields + ('resources',)

Now in MyKubernetesPodOperator the resources parameter is templated and you can use Jinja syntax with it.

Elad Kalif
  • 14,110
  • 2
  • 17
  • 49
  • But I think as of airflow version 2.1.4 developments, I can't use dict for `resources` parameter as it initialises the operator with constructor value as str (not evaluated as template strings happen at DAG's runtime) -> Error received is as `airflow.exceptions.AirflowException: Expected , got ` – Amol.D Mar 31 '22 at 12:45
  • in newer cncf provider resources accept V1ResourceRequirements not dict. This answer was written several months ago.. – Elad Kalif Mar 31 '22 at 12:58
  • It doesn't make sense to template V1ResourceRequirements object. If you have such requirement then you need to inherit from the KubernetesPodOperator as I shown and have some other parameters which you will template and then use them to construct V1ResourceRequirements object. – Elad Kalif Mar 31 '22 at 13:01
  • Yes, inheriting and overloading the `KubernetesPodOperator` constructor is what I don't want to do as it adds lots of maintenance to the code when Airflow plans/changes the KubernetesPodOperator implementation. – Amol.D Apr 05 '22 at 04:54
  • @Amol.D as explained you can not template `V1ResourceRequirements` directly. You will have to mantine custom code where DAG author pass the parameters with dict then you will template the dict and after that assign the relevant tempalted parameters in `V1ResourceRequirements`. Only then you can invoke the operator. – Elad Kalif Apr 05 '22 at 06:40
  • Thanks @Elad. Yeah, I understood it. But the best practice to use Airflow is to have minimum maintenance code. – Amol.D Apr 05 '22 at 11:50
  • @Amol.D I don't know where you got it from... Not something written in Airflow official docs. Should the problem explained here is a requirement you need to resolve - there is no alternative that I'm aware. – Elad Kalif Apr 05 '22 at 12:22