airflow2.0 with KubernetesPodOperator: TemplateNotFound

Question

I am using Airflow2.0 with KubernetesPodOperator want to run a command that use as a parameter a file from inside the image ran by the Operator. This is what I used:

KubernetesPodOperator(
    namespace=commons.kubernetes_namespace,
    labels=commons.labels,
    image=f"myregistry.io/myimage:{config['IMAGE_TAG']}",
    arguments=[
        "python",
        "run_module.py ",
        "-i",
        f'args/{config["INPUT_DIR"]}/{task_id}.json'
    ],
    name=dag_name + task_id,
    task_id=task_id,
    secrets=[secret_volume]
)

But this gives me the error:

raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: args/airflow2test/processing-pipeline.json

The image does not use any macros.

Anyone has any clue? What do I do wrong?

Answer 1

This was a bug that started with PR released in version 2.0.0 of apache-airflow-providers-cncf-kubernetes. The goal of the change was to allow templating of .json files. There was a GitHub issue about the problems it created. The bug was eventually resolved by PR which was released in version 2.0.2 of the provider.

Solution:

1. Upgrade to the latest apache-airflow-providers-cncf-kubernetes (currently 2.0.2)
2. If upgrade is not an option use custom KubernetesPodOperator

There are two ways to workaround that problem one is to change template_fields the other is to change template_ext:

1st option: As posted on issue by raphaelauv is not to allow rendering of arguments field:

class MyKubernetesPodOperator(KubernetesPodOperator):
    template_fields = tuple(x for x in KubernetesPodOperator.template_fields if x != "arguments")

2st option: if you prefer not to render .json files:

class MyKubernetesPodOperator(KubernetesPodOperator):
    template_ext = ('.yaml', '.yml',)