-1

I am new to airflow, I want to call to an external licensing server every time a DAG run is successfully completed.

vajr1
  • 27
  • 5
  • could you provide some more information in your question? What do you mean by "call" - are you asking about making a HTTP request to another service? Are you able to share examples of your code, things you have tried etc. – Josh Sep 10 '21 at 04:47
  • Yes its a HTTP request to another service. The server would return the info about licensing and all . I couldn't find any resources online for the same. Thanks – vajr1 Sep 10 '21 at 07:12
  • Please provide enough code so others can better understand or reproduce the problem. – Community Sep 18 '21 at 00:42

1 Answers1

1

There is no "successful completion" event in Airflow. You can add Cluster Policies - https://airflow.apache.org/docs/apache-airflow/stable/concepts/cluster-policies.html for "before start" events (for DAGs and Tasks.

I think the only way to make sure it works is to have a Task as the "last" step in your DAG (when all other succeed) with "on_success_callback" set.

It might be impossible to add such task automatically if you would like that.

You could try to utilise "Task instance Mutation" for that, but I think you cannot change callbacks and you'd have to know which task you'd need to mutate somehow.

https://airflow.apache.org/docs/apache-airflow/stable/concepts/cluster-policies.html#task-instance-mutation

One thing that you COULD do if you want to somehow "force" existence of such task and a callback, you could add "Cluster Policy" that could read your DAG and check that such task exists (and do not allow the DAG to run without it). But that's kinda complex I think.

Jarek Potiuk
  • 19,317
  • 2
  • 60
  • 61