Unexpected Jinja Template Behaviour in Custom Airflow Operator

Question

I have made a custom sensor in Airflow which inherits BashSensor.

Sensor :

class MySensor(BashSensor):
    def __init__(self, time, **kwargs):  # {{ ts }} is passed as time in the DAG
        self.time = time
        cmd = f"java some-other-stuff {self.time}"  # rendered/correct value for self.time
        super().__init__(**kwargs, bash_command=cmd)

    def poke(self, context):
        status = super().poke()  # returns True or False
        if status:
            print(self.time)  # {{ ts }} is printed instead of rendered value
        else:
            print("trying again")
        return status

When I look at the rendered tab for the operator task in DAG I see bash_command has the correct rendered value ({{ ts }} is passed as time).

The problem is whenever poke is called and True is returned, I see {{ ts }} in the print statement instead of the rendered value.

I expect self.time to have the rendered value (some timestamp) not {{ ts }} when I print it in poke function.

Elad Kalif · Accepted Answer · 2022-11-14T13:14:01.600

1

Both cmd and time are not templated field in your code so Jinja engine does not handle them. The reason you see the command being templated is because in the super call you do:

bash_command=cmd

and bash_command is templated field of BashSensor So while the command is parsed to the correct string as expected the individual components that created it does not contain the render value. To explain in some more details: time = "{{ ds }}" will always stay as this string. it will never be rendered.

When you do cmd = f"java some-other-stuff {self.time}" it becomes:

"java some-other-stuff {{ ds }}"

This string is assigned to bash_command which is templated field and when the code is executed the value of {{ ds }} is rendered.

To solve your issue you can simply add the parameters you want to template to the sequence:

class MySensor(BashSensor):
    ...
    template_fields: Sequence[str] = tuple({'time'} | set(BashSensor.template_fields))
    ...

edited Nov 14 '22 at 13:14

answered Nov 13 '22 at 12:08

Elad Kalif

14,110
2
17
49

I had `template_fields = ('time',)` in my sensor. I don't remember the exact problem I faced but even the `bash_command` had `{{ ts }}` instead of value. Removing `template_fields` fixed it. I will test it out and post an update here. – Dark Matter Nov 13 '22 at 12:35
This is not consistent with the code you posted in the question. Please post the actual code and the traceback otherwise its hard to assist. Noting that in the comment you override the value of templated gields which is wrong resulting in bash command not being templated. – Elad Kalif Nov 13 '22 at 16:13
After using the `template_fields` statement you provided both values rendered correctly. The problem was I was overriding the `template_fields` of `BashSensor` as you pointed out. Shouldn't it be `template_fields: Sequence[str] = ...` instead of `template_fields = template_fields: Sequence[str] = ...` in your code above? – Dark Matter Nov 14 '22 at 12:57
yes that is a copy paste typo. fixed – Elad Kalif Nov 14 '22 at 13:14

Unexpected Jinja Template Behaviour in Custom Airflow Operator

1 Answers1