0

i have more dags (say 150dags) in my airflow account. My airflow version = 2.4.0

im using url and query parameter http://localhost:8081/api/v1/dags?limit=1000&offset=0

But my response consists only 100 records(dags). i think query parameters are not working here.

How can i fetch all dags available inside my airflow.

vbala vbala
  • 15
  • 3

1 Answers1

0

there is a PR that explain that the limit can not be passed and if you put more then the limit the fallback would be the limit

if you want to change the limit in the api, you can change "maximum_page_limit" in airflow.cfg to other number (default = 100)

Another option is to play with offset until you do not get dags in the list. for example : first call limit=100, offset=0, second call limit=100, offset=101 and so on until empty response.

http://localhost:8081/api/v1/dags?limit=100&offset=0
http://localhost:8081/api/v1/dags?limit=100&offset=101

also, an option not in the api is to create a dag with a task and using DagBag to get all dags details.

in this example I print all the dag ids

from datetime import datetime

from airflow import DAG, settings
from airflow.decorators import task
from airflow.models import DagBag

with DAG(
        dag_id="test_dag",
        schedule_interval=None,
        default_args={
            "start_date": datetime(2022, 1, 1),
            "retries": 0,
            "catchup": False,
        },
        render_template_as_native_obj=True,
        tags=["test"],
) as dag:
    dag.doc_md = __doc__


    @task
    def print_dags():
        dagbag = DagBag(settings.DAGS_FOLDER)
        print(dagbag.dags.keys())


    (print_dags())
ozs
  • 3,051
  • 1
  • 10
  • 19
  • ok, But if im already having multiple dags , how can i able to get all dags through rest api. – vbala vbala Jun 29 '23 at 11:01
  • @vbalavbala, what is the total_entries you see in the json ? also are all active? by default only_active=True – ozs Jun 29 '23 at 11:26
  • in response json, i have 100 dags details under "dags" key and under "total_entries" key i have value 110, Also in my account i have 110 dags. all dags are active dags. – vbala vbala Jun 29 '23 at 11:36
  • update my answer, there is parameter "maximum_page_limit" in airflow.cfg that can be changed – ozs Jun 29 '23 at 11:51
  • OK , Other than this , is there any other way for this approach? – vbala vbala Jun 29 '23 at 11:56
  • I think there is, play with the offset by making few calls. updated my answer – ozs Jun 29 '23 at 12:02
  • suppose if im having 150 dags , how can i make use of offset &limit query parameter. give some example for this. – vbala vbala Jul 03 '23 at 04:08
  • @vbalavbala I updated my answer with an example of to api calls.of course you can write a loop until no items in the response – ozs Jul 03 '23 at 04:24