0

we have created a task for sensor operation, but the task name will be dynamically updated. i.e., f"{table_name}_s3_exists". We have a scenario where we have to check a table's location twice, but if the task is still present, we don't have to create the sensor. Is there a way to find whether the task exists or not within the DAG during building ?

Santhosh
  • 71
  • 1
  • 5

2 Answers2

0

The CLI command

airflow tasks list [-h] [-S SUBDIR] [-t] [-v] dag_id

will give you list of all the dags.

https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#list_repeat6

You can also use the REST API to get the same info: https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/get_tasks

FraBreezy
  • 1
  • 1
  • thanks for the information, I was looking more of a programmatic way of getting the status/state of a task to check whether it exists or not. – Santhosh Apr 07 '22 at 13:53
0

You could try the get_tasks endpoint in the Airflow REST API. The endpoint returns a lot of information for tasks in a given DAG.

Josh Fell
  • 2,959
  • 1
  • 4
  • 15