0

I need to have an API call from the Ariflow dag.

Below is the sample json_string I use in code:

    def api_call_func(**context):
        source_file_name = context["dag_run"].conf["source_file_names"]
    json_data = { "filenames":[
                 {"FileName":[f'Source_credit_{{{{ ts_nodash }}}}']}
                  ]
                }

        json_string = json.dumps(json_data, 
                        skipkeys = True, 
                        allow_nan = True)
        requests.post(url = API_URL, data = json_string)

    api_task = PythonOperator(
    task_id='api_task',
    provide_context=True,
    python_callable=api_call_func,
    dag=dag,
    )

but it is resulting in the below response:

<filenames>
         <FileName>f'Source_credit_{{ ts_nodash }}</FileName>
</filenames>

below is the response desired with the currentDateTime:

<filenames>
         <FileName>f'Source_credit_20210408010223</FileName> 
</filenames>

How to pass the ts_nodash macros in json? or how to pass the dag's execution datetime in json?

Rv R
  • 191
  • 4
  • 14
  • 1
    How do you exactly use `json_str`? Templates are processed only if the string containing them is passed as an argument to a templated parameter of an operator. E.g., `op_args` parameter of the `PythonOperator`. – SergiyKolesnikov Apr 08 '21 at 18:00
  • As was mentioned Airflow defines which parameter can be templated or not and it strongly depends on the particular operator implementation. Can you share some code lines to us to proper get the point? – Nick_Kh Apr 09 '21 at 07:44
  • Please find the code lines added. hope it helps @Nick_Kh – Rv R Apr 10 '21 at 16:07
  • I have added code lines which shows how json_str is used. please help out @SergiyKolesnikov – Rv R Apr 11 '21 at 13:31

1 Answers1

2

Templates are processed only if the string containing them is passed as an argument to a templated parameter of an operator. E.g., op_args parameter of PythonOperator.

In your case, the value of ts_nodash is passed to api_call_func() as an argument by the PythonOperator. So, you can use the context parameter to access it:

json_data = {
    "ID": "A001-001",
    "SourceName": sourcefile,
    "filenames": [{"FileName": [f"Source_credit_{context['ts_nodash']}"]}],
}

Note, the values for the rest of the default variables can be accessed in the same way.

SergiyKolesnikov
  • 7,369
  • 2
  • 26
  • 47
  • can I also do - `json_data = { "ID": "A001-001", "SourceName": sourcefile, "filenames": [{"FileName": [f"Source_credit_{ context['execution_date.subtract(seconds=2).strftime("%Y%m%d%H%M%S")'] }"]}], } ` @SergiyKolesnikov – Rv R Apr 12 '21 at 05:43
  • I don't think so, but you can get `cotext['execution_date']` then convert it to datetime and process it further. – SergiyKolesnikov Apr 12 '21 at 06:36
  • 1
    `{ context["execution_date"].subtract(seconds=2).strftime("%Y%m%d%H%M%S") }` this one worked. Thanks a lot :) @SergiyKolesnikov – Rv R Apr 12 '21 at 07:49