0

I have two dynamically generated tasks task_1 and task_2. task_2 depends on task_1. Both are instances of KubernetesPodOperator.

If an empty map (0 length) is passed to task_2, instead of skipping task_2 per the documentation, task_2 is put in a removed state and the pipeline fails.

I am running Airflow 2.5.3.

Here is the code that reproduces the problem.

from airflow.decorators import dag, task
import pendulum

@dag(
    dag_id='test_dynamic_kpo_simple',
    schedule=None,
    start_date=pendulum.datetime(2023, 7, 27, tz="UTC")
)
def test_dynamic_kpo_simple():

    @task.kubernetes(
        image="python:3.8-slim-buster",
        name="k8s_test",
        namespace="default",
        in_cluster=False
    )
    def tk1(arg: int):
        import time

        print(f"Hello: {arg}")


    @task.kubernetes(
        image="python:3.8-slim-buster",
        name="k8s_test",
        namespace="default",
        in_cluster=False
    )
    def tk2(arg: int):
        import time

        print(f"Hello again from tk2: {arg}")


    @task
    def t3():
        print('I am here ===========')

    task_1 = tk1.expand(arg=[1, 2])
    task_2 = tk2.expand(arg=[])
    task_3 = t3()

    task_1 >> task_2 >> task_3
        


test_dynamic_kpo_simple()

Here is the Web UI screenshot

Airflow Web UI

Hui
  • 97
  • 7

0 Answers0