0

I want to first get a dic from one operator, and then iterate the result and run another operator. The dictionary contains dataset and tables which contain words like "daily" "weekly" or "monthly".

The script is as follows:

@task
def generate_dic_task():
#code ....
    return dic

with DAG(dag_id='test', default_args=default_args, schedule_interval='@daily') as dag:

    dic = generate_dic_task()

    for dataset in dic:
        
        if "monthly" in dataset:
            limit_date = (datetime.now() - timedelta(days=93)).strftime("%Y-%m-%d")
        elif "weekly" in dataset:
            limit_date = (datetime.now() - timedelta(days=22)).strftime("%Y-%m-%d")
        elif "daily" in dataset:
            limit_date = (datetime.now() - timedelta(days=4)).strftime("%Y-%m-%d")
        else:
            limit_date = datetime.now().strftime("%Y-%m-%d")

        for table in dic[dataset]:
            
            run_op = DbtRunOperator(
                            task_id='run_op_{}_{}'.format(dataset, table),
                            vars={
                                'table_name': table,
                                'dataset': dataset,
                                'limit_date': limit_date
                            
                            },
                             dag=dag
            )

Currently, I have an error because Airflow don't understand dic = generate_dic_task()

I have try a lot of things : I have put the dbtrunoperator in an other @task but didn't work. I have also try xcom_pull and xcom_push but it doesn't work also.

I am expecting that at the end my dag dbtrunoperator will run for every variables that contains the dictionnary.

Thank for your help !

RubTest
  • 21
  • 3

0 Answers0