Running airflow tasks/dags in parallel

Question

I'm using airflow to orchestrate some python scripts. I have a "main" dag from which several subdags are run. My main dag is supposed to run according to the following overview:

I've managed to get to this structure in my main dag by using the following lines:

etl_internal_sub_dag1 >> etl_internal_sub_dag2 >> etl_internal_sub_dag3
etl_internal_sub_dag3 >> etl_adzuna_sub_dag
etl_internal_sub_dag3 >> etl_adwords_sub_dag
etl_internal_sub_dag3 >> etl_facebook_sub_dag
etl_internal_sub_dag3 >> etl_pagespeed_sub_dag

etl_adzuna_sub_dag >> etl_combine_sub_dag
etl_adwords_sub_dag >> etl_combine_sub_dag
etl_facebook_sub_dag >> etl_combine_sub_dag
etl_pagespeed_sub_dag >> etl_combine_sub_dag

What I want airflow to do is to first run the etl_internal_sub_dag1 then the etl_internal_sub_dag2 and then the etl_internal_sub_dag3. When the etl_internal_sub_dag3 is finished I want etl_adzuna_sub_dag, etl_adwords_sub_dag, etl_facebook_sub_dag, and etl_pagespeed_sub_dag to run in parallel. Finally, when these last four scripts are finished, I want the etl_combine_sub_dag to run.

However, when I run the main dag, etl_adzuna_sub_dag, etl_adwords_sub_dag, etl_facebook_sub_dag, and etl_pagespeed_sub_dag are run one by one and not in parallel.

Question: How do I make sure that the scripts etl_adzuna_sub_dag, etl_adwords_sub_dag, etl_facebook_sub_dag, and etl_pagespeed_sub_dag are run in parallel?

Edit: My default_args and DAG look like this:

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': start_date,
    'end_date': end_date,
    'email': ['myname@gmail.com'],
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 0,
    'retry_delay': timedelta(minutes=5),
}

DAG_NAME = 'main_dag'

dag = DAG(DAG_NAME, default_args=default_args, catchup = False)

Mr. President, can you edit your post and add what your `DAG(...)` and `default_args` looks like. — Mendhak, Oct 10 '18 at 13:41

kaxil · Accepted Answer · 2018-10-10T13:57:35.050

35

You will need to use LocalExecutor.

Check your configs (airflow.cfg), you might be using SequentialExectuor which executes tasks serially.

Airflow uses a Backend database to store metadata. Check your airflow.cfg file and look for executor keyword. By default, Airflow uses SequentialExecutor which would execute task sequentially no matter what. So to allow Airflow to run tasks in Parallel you will need to create a database in Postges or MySQL and configure it in airflow.cfg (sql_alchemy_conn param) and then change your executor to LocalExecutor in airflow.cfg and then run airflow initdb.

Note that for using LocalExecutor you would need to use Postgres or MySQL instead of SQLite as a backend database.

More info: https://airflow.incubator.apache.org/howto/initialize-database.html

If you want to take a real test drive of Airflow, you should consider setting up a real database backend and switching to the LocalExecutor. As Airflow was built to interact with its metadata using the great SqlAlchemy library, you should be able to use any database backend supported as a SqlAlchemy backend. We recommend using MySQL or Postgres.

edited Oct 10 '18 at 13:57

answered Oct 10 '18 at 13:41

kaxil

17,706
2
59
78

Thank you for your answer! Could you elaborate a bit more on what you mean? The subdags are using a mysql database but I'm not sure whether that's what you mean. Can I just change `executor = SequentialExecutor` to `executor = LocalExecuter`? – Mr. President Oct 10 '18 at 13:52
2

Not subdags. Airflow uses a Backend database to store metadata. Check your `airflow.cfg` file and look for `executor` keyword. By default Airflow uses `SequentialExecutor` which would execute task sequentially no matter what. So to allow Airflow to run tasks in Parallel you will need to create a database in Postges or MySQL and configure it in `airflow.cfg` (`sql_alchemy_conn` param) and then change your executor to `LocalExecutor`. – kaxil Oct 10 '18 at 13:56
Allright, I think I understand what you mean! Isn't there a way to do so without creating a new database though? And what's the reason that it isn't possible with the default sqlite setup? – Mr. President Oct 10 '18 at 14:14
And one more thing; could you show me where you read that you need to use mysql or postgres in order to use `LocaExecutor`? Because I can't find that anywhere :) – Mr. President Oct 10 '18 at 14:25
7

SQLite only supports 1 connection at a time. It doesn't support more than 1 connection. Hence, you need to use a different database like Postgres or MySQL. https://airflow.incubator.apache.org/howto/initialize-database.html - Check this link where we recommend using it. I am one of the committers to the Airflow project and based on my experience Postgres would be the best option. :) – kaxil Oct 10 '18 at 14:29

score 10 · Answer 2 · answered Apr 06 '22 at 07:40

10

One simple solution to run tasks in parallel is to put them in [ ] brackets. For example : task_start >> [task_get_users, task_get_posts, task_get_comments, task_get_todos]

For more information you can read this Article from towardsdatascience

answered Apr 06 '22 at 07:40

Luca

131
1
3

Thanks for this info! Now Medium / TowardsDataScience is behind a real pay wall. So we shouldn't provide links to it any more. – Mike Williamson Jul 27 '23 at 15:58

score 5 · Answer 3 · edited Sep 19 '21 at 19:46

5

Try:

etl_internal_sub_dag3 >> [etl_adzuna_sub_dag, etl_adwords_sub_dag, etl_facebook_sub_dag, etl_pagespeed_sub_dag]

[etl_adzuna_sub_dag, etl_adwords_sub_dag, etl_facebook_sub_dag, etl_pagespeed_sub_dag] >> etl_combine_sub_dag

edited Sep 19 '21 at 19:46

double-beep

5,031
17
33
41

answered Sep 08 '21 at 22:24

LeComp

51
1
1

Please provide additional details in your answer. As it's currently written, it's hard to understand your solution. – Community Sep 08 '21 at 22:32

Running airflow tasks/dags in parallel

3 Answers3

Linked