Accessing value of {{ts_nodash}} in airflow

Question

I need to access the value of {{ts_nodash}} to format it as a datetime and subtract an hour from it. But this gives me an error on dag saying: ValueError: time data '{{ ts_nodash }}' does not match format '%Y%m%dT%H%M%S' This is how my method looks like:

          ts_nodash = datetime.strptime("{{ ts_nodash }}", "%Y%m%dT%H%M%S")
          get_numbers = gke_wrapper.execute_gke_operator(
                task_id=f"get_numbers_{ctype_name}",
                labels=get_cost_labels(
                    pod=Pod.pod,
                    service="service",
                    id="mmoe",
                    environment="dev",
                    component="airflow",
                ),
                cmd=["sh"],
                args=[
                    "get_numbers.sh",
                    output_table_name,
                    dump_table_name,
                    model_path,
                    model_path,
                    ctype_name,
                    config.variants[ctype_name],
                    (ts_nodash - timedelta(hours=config.delay_interval)).strftime(
                        "%Y%m%dT%H%M%S"
                    ),
                ],
                nodepool=_NODE_POOL,
                image=_IMAGE,
                resource=_RESOURCES,
            )

Can someone mention how can I access the template variable value? I tried many different ways of passing {{ts_nodash}} within args=[] but apparently the datetime parsing operation occurs before the template value is made available. Any help will be much appreciated.

Just use `dag_run.logical_date`, which is a `datetime`, rather than trying to parse the `ts` string. — 0x5453, Jul 20 '23 at 14:06
Can you explain how it'll look in code? Sorry I'm new to airflow — Himanshu Doi, Jul 20 '23 at 14:21
Based on what I read, dag_run.logical_date also needs to be accessed as a template, which is same as ts_nodash. The problem to access it as a variable still persists — Himanshu Doi, Jul 20 '23 at 14:32

score 1 · Answer 1 · answered Jul 21 '23 at 07:39

Templates evaluated during the Dag run (i.e. inside the execute method of the operator). Looks like you are trying to fetch ts_nodash outside the run time of the dag.

Below is an example of using templates:

op = BashOperator(
    task_id="pre-validation",
    bash_command="echo 'this is ts_nodash {{ ts_nodash }}'\n"
                 "echo 'this is dag_run.logical_date {{ dag_run.logical_date }}'",
    dag=dag,
)

So you will have to pass the template itself to the operator like below and it will be evaluated during the run time:

...
args = [
    "get_numbers.sh",
    output_table_name,
    dump_table_name,
    model_path,
    model_path,
    ctype_name,
    config.variants[ctype_name],
    "{{ ts_nodash }}"
],
...

You can also find some additional context on [template variables](https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html) in the docs along with some more documentation on [jinja templating](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/operators.html#jinja-templating). — RNHTTR, Jul 29 '23 at 02:21

Accessing value of {{ts_nodash}} in airflow

1 Answers1