0

I've been using the airflow pool to control my concurrent tasks. so I've created a test_pool with 10 slots and have created 4 tasks, out of which I have assigned 2 tasks with more priority weight by filling all the 10 slots each. However, the tasks are ignoring priority weights and being triggered weirdly. Here's my dag code

with DAG(
    DAG_ID,
    schedule_interval=dag_run_schedule_interval(DAG_ID),
    default_args={
        'owner': OWNER,
        'depends_on_past': False,
        'start_date': dag_start_date("daily"),
        'email_on_failure': False,
        'email_on_retry': False,
        'pool': 'test_pool',
    },
) as dag:

    def go_to_sleep():
        time.sleep(30)
    
    task_x = PythonOperator(
        task_id='task_x',
        python_callable=go_to_sleep,
        priority_weight=5,
        pool_slots=10
    )

    task_y = PythonOperator(
        task_id='task_y',
        python_callable=go_to_sleep,
        priority_weight=5,
        pool_slots=10
    )

    task_z = PythonOperator(
        task_id='task_z',
        python_callable=go_to_sleep,
        priority_weight=2,
        pool_slots=10
    )

    task_a = PythonOperator(
        task_id='task_a',
        python_callable=go_to_sleep,
        priority_weight=2,
        pool_slots=10
    )

    task_x >> task_y
    task_z >> task_a

The expectation would be the task_x to trigger first and then task_y as the priority_weights of these two tasks are higher than task_a and task_z.

However, it is triggering in this order task_x, task_z, task_y, task_a. Did I miss something?

Tula
  • 307
  • 6
  • 15

2 Answers2

0

pool priority works like 1 , 2 ,3, and you are directly giving the 5 instead of 3 if you want to set a precedent follow the proper rules, Here a link that can help you understand this https://www.astronomer.io/guides/airflow-pools/

Mobeen
  • 76
  • 9
0

I've added 'weight_rule': 'absolute', in my default args, and it is working as expected. The default value for this is downstream.

Tula
  • 307
  • 6
  • 15