I have two dynamically generated tasks task_1 and task_2. task_2 depends on task_1. Both are instances of KubernetesPodOperator.
If an empty map (0 length) is passed to task_2, instead of skipping task_2 per the documentation, task_2 is put in a removed state and the pipeline fails.
I am running Airflow 2.5.3.
Here is the code that reproduces the problem.
from airflow.decorators import dag, task
import pendulum
@dag(
dag_id='test_dynamic_kpo_simple',
schedule=None,
start_date=pendulum.datetime(2023, 7, 27, tz="UTC")
)
def test_dynamic_kpo_simple():
@task.kubernetes(
image="python:3.8-slim-buster",
name="k8s_test",
namespace="default",
in_cluster=False
)
def tk1(arg: int):
import time
print(f"Hello: {arg}")
@task.kubernetes(
image="python:3.8-slim-buster",
name="k8s_test",
namespace="default",
in_cluster=False
)
def tk2(arg: int):
import time
print(f"Hello again from tk2: {arg}")
@task
def t3():
print('I am here ===========')
task_1 = tk1.expand(arg=[1, 2])
task_2 = tk2.expand(arg=[])
task_3 = t3()
task_1 >> task_2 >> task_3
test_dynamic_kpo_simple()
Here is the Web UI screenshot