0

I created a collection of dags dynamically (using the same .py for all). And there is one build-DAG that I cannot run :

airflow.exceptions.AirflowException: dag_id could not be found: `build-DAG`. Either the dag did not exist or it failed to parse.
at get_dag (/usr/local/lib/python2.7/site-    packages/airflow/bin/cli.py:130)
at run (/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py:353)
at <module> (/usr/local/bin/airflow:27)

For this dag I can see previous logs, code and all the stuff in the UI but I can't run it. Any idea how to debug this? could be useful to restart composer instances?

The rest of the dags dynamically created works fine.

I'm using something similar to this in order to create the dags: https://gist.github.com/tmarthal/edeae7f6f8780dc53887a16b7b20f205

Thanks in advice. Edu

Update: I'm using composer-0.5.1-airflow-1.9.0

Update August 2, 2018 : I migrated to composer-1.0.0-airflow-1.9.0 and still happen

edduuar
  • 21
  • 4
  • Is there anything different about that DAG's generation? You should ensure that it is in the globals of the module. Try importing the module in a python shell and ensuring that you can access that DAG. – Trevor Edwards Aug 28 '18 at 15:40

1 Answers1

0

This wasn't an airflow issue, was a concurrency issue.

I have two backends returning the list of dynamics dags, and each backend has different DAGs id list. When the DAG definition used Backend A, create 20 DAGs and when used Backend B create only 18 DAGs. Then airflow intermittently fails when I tried to run the DAG 19th. My solution was synced both backend.

Regards Eduardo

edduuar
  • 21
  • 4