0

I need a scheduled task in airflow to run with different parameters. One way would be to write different dags for different parameters, but i was wondering if there is a better way of doing this like how we pass parameters to manual trigger.

Mahdi Sadeghi
  • 673
  • 5
  • 13
  • can you provide an example of the schedule you want the DAG to run on, and example of the different parameters you'd want for each run? – Josh Aug 17 '21 at 08:59
  • for example i have a module which collects a company data from an api. given a different base_url i can use the same module for collecting another companies data (they are using the same product and have exact same apis, just different urls). now if i make this url a parameter to dag, the same dag can be used for any company i want to collect data. but this is the case for manual trigger, i need this dag to run at 1 am for Company A and at 2 am for company B every day. – Mahdi Sadeghi Aug 17 '21 at 10:54
  • Check [this answer](https://stackoverflow.com/a/68107775/10569220) for examples on how to pass params to a DAG. I think it's what you are looking for. – NicoE Aug 17 '21 at 12:13

1 Answers1

2

The bigger issue with writing the same DAG with different URLs is that you're breaking the DRY (Don't Repeat Yourself) principle. If you need different schedules and you don't want to repeat the code twice, you can take this DAG factory idea and build your own factory for these two DAGs. You'll finally have two DAG files that invokes the same factory with the parameters of schedule (1 AM and 2 AM) and URL.

bruno-uy
  • 1,647
  • 12
  • 20