1

Airflow UI comes very handy to check the status of Airflow DAG progress.

Airflow Rest API is another way to check the status to Airflow DAGs but it requires authentication token.

We can get authentication token from Airflow UI, but if Airflow UI is down it would be difficult to get authentication token and Airflow DAG status.

Is there any other way to check/monitor and clear task instance from backend (apart from Rest API and Airflow UI) ?

Khilesh Chauhan
  • 739
  • 1
  • 10
  • 36
  • Why would it be difficult to authenticate? You can do all of that even when webserver is down. The Rest API is not related to the webserver. – Elad Kalif May 28 '22 at 16:13
  • Because its LDAP authentication not the Airflow UI created Users. – Khilesh Chauhan May 28 '22 at 16:15
  • Are you trying to create a monitoring solution or your use case is a specific user would like to know the status of a specific DAG just for the cases where webserver is down? – Elad Kalif May 28 '22 at 16:18
  • Just to ease monitoring DAGs status, when webserver is down – Khilesh Chauhan May 28 '22 at 16:25
  • this is not exactly what you are looking for but why dont you set email notification for both fail and success status? would this be an indirect solution to what you are looking for? – memo May 28 '22 at 16:42
  • No, it wont serve the purpose. By DAG status, i would like to know individual task status, this will help if a task sensor is waiting for something. Email notification will only let failure/success status of entire DAG. – Khilesh Chauhan May 28 '22 at 16:46

2 Answers2

1

We have setup status checks on airflow health check endpoint based on this Airflow doc page Checking Airflow Health Status. We have serverless functions running every 5 minutes to check that the status for metabase and scheduler are healthy.

When Airflow is down, you can get alerts routed directly to Slack channel / Email / Opsgenie Alerts through another code block defined in the serverless function.

Vinay Kulkarni
  • 300
  • 1
  • 5
  • 13
0
from airflow.models.dagrun import DagRun
from airflow.utils.state import DagRunState

dag_runs = DagRun.find(dag_id='the_dag_id_you_want_to_check')
last_run = dag_runs[-1]
print('the dag state is -->: ', last_run.state)
Afshin Amiri
  • 3,438
  • 1
  • 20
  • 21