0

I have a parent dag that is used to trigger another. When I trigger the child dag, the child dags is only queued but never running.

This is my parent_dag.py file

from airflow import DAG
from datetime import datetime, timedelta

from airflow.decorators import dag
from airflow.operators.trigger_dagrun import TriggerDagRunOperator
from airflow.operators.python import PythonOperator


def setup_environment(ti, **context):
    print("parent dag")


with DAG(
    dag_id="master",
    default_args={"owner": "airflow"},
    start_date=datetime(2022, 8, 8),
    schedule_interval=None,
    tags=["master"],
    catchup=False,
) as dag:

    setup = PythonOperator(
        task_id="setup",
        python_callable=setup_environment,
        provide_context=True,
    )

    dag_tabular = TriggerDagRunOperator(
        task_id="trigger_tabular",
        trigger_dag_id="dag_tabular",
        trigger_run_id="{{run_id}}",
        wait_for_completion=True,
    )
setup >> dag_tabular

and this is my tabular.py file

from airflow import DAG
from datetime import datetime, timedelta
from airflow.operators.python import PythonOperator
import time


def setup_environment(ti, **context):
    print("tabular dag")


def stopper(ti, **context):
    time.sleep(60)
    print("done with tabular")


with DAG(
    dag_id="dag_tabular",
    default_args={"owner": "airflow"},
    start_date=datetime(2022, 8, 8),
    schedule_interval=None,
    tags=["tabular"],
    catchup=False,
) as dag:
    setup = PythonOperator(
        task_id="setup", python_callable=setup_environment, provide_context=True
    )
    stopper = PythonOperator(
        task_id="stop_tabular", python_callable=stopper, provide_context=True
    )

setup >> stopper

If i look in the UI, dag_tabular is queued but not running. If I remove wait_for_completion, both the dags run.

So I am guessing that there is some lock situation. The parent dag is waiting for tabular dag to complete but tabular dag is not able to acquire the resources (a worker) to complete it's task.

Or maybe it's something much simpler.

I looked at the following parameters in the configuration file but could not find anything:

# Number of workers to run the Gunicorn web server
workers = 4

worker_concurrency = 16

max_active_runs_per_dag = 16

Edit:

I ran these dags on Google Cloud Compose and they are working just as expected that is, dag_tabular gets triggered and then runs. So I am guessing that there must be some issue with the number of workers.

shubh gupta
  • 94
  • 7
  • Given `dag_tabular` is not enabled by default, did you go into the UI and unpause the DAG? – Jorrick Sleijster Oct 07 '22 at 08:08
  • Yes, dag_tabular was unpaused before running the parent dag. I ran these dags on google cloud compose and everything is working fine. – shubh gupta Oct 07 '22 at 08:43
  • Just to verify, is the DAGRun being queued(the bars the top of the grid view) or are the TaskInstance being queued(the squares in the grid view)? – Jorrick Sleijster Oct 07 '22 at 09:21
  • 1
    Tasks are not being queued. The dagRun is queued. – shubh gupta Oct 07 '22 at 09:30
  • Then you should search the problem in either your DAG definition or in the schedulers. The workers have nothing to do with it then :). – Jorrick Sleijster Oct 10 '22 at 09:30
  • Is it possible you have earlier DagRuns that are still running that prevent this one from running? (as you might have reached the maximum of active parallel DAGRuns) – Jorrick Sleijster Oct 10 '22 at 09:31
  • No. There was only the parent dag running and the child dag was queued. I think it was due to some airflow bug. I switched to version 2.3.3 and now everything is runnning as expected. – shubh gupta Oct 11 '22 at 06:44

0 Answers0