0

I am trying to push parameter with dict inside from airflow xcom_pull to PapermillOperator like that:

send_to_jupyter_operator = PapermillOperator(
     task_id='send_to_jupyter',     
     input_nb="./dags/notebooks/input_test.ipynb",     
     output_nb="./dags/notebooks/{{ execution_date }}-result.ipynb",     
     parameters={"table_list": "{{ ti.xcom_pull(dag_id='select_data_from_table',task_ids='select_data', key='table_result_dict') }}"} )

Task with task_id='select_data' - its a PythonOperator which push dict to xcom.

Inside ti.xcom_pull(dag_id='select_data_from_table', task_ids='select_data', key='table_result_dict') - dict of dicts (keys - name of dimension, values - dicts with key = attribute name, values - list of values);

But with this syntax jupyter-notebook import string, not dict, like: table_list = "{'key1': {'attr1': []}}"

Are there any tips to solve this problem?

I have already tried to use: parameters={"table_list": {{ ti.xcom_pull(dag_id='select_data_from_table', task_ids='select_data', key='table_result_dict') }} } - in this keys python doesn't know what 'ti' is actually.

parameters={"table_list": {{ context['ti'].xcom_pull(dag_id='select_data_from_table', task_ids='select_data', key='table_result_dict') }} } - in this keys python doesn't know what 'context' is actually.

VorobevNikita
  • 1
  • 1

1 Answers1

0

I have resolved problem with another way.

Just add this to your jupyter-notebook:

list = json.loads(input_list.replace("\'",'"').replace('None', 'null'))
VorobevNikita
  • 1
  • 1