How to set an alert for Azure Data factory when Pipeline takes more than N minutes to complete

Question

I need to setup an alert system if my Azure Datafactory pipeline runs for more than 20 minutes. The alert should come while the pipeline is running and the duration passes 20mins, not after the completion of pipeline. How can I do this? I think this can be done using Azure function but I am not familiar with it so I'm in search for a script for the same.

Answer 1

Yes, azure function is a solution to acheive your requirement.

For example, if you are using Python. You need an azure function that runs periodically to monitor the status of the pipeline. The key is the duration time of the pipeline. pipeline is based on activities. You can monitor every activity.

In Python, This is how to get the activity you want:

https://learn.microsoft.com/en-us/python/api/azure-mgmt-datafactory/azure.mgmt.datafactory.operations.activityrunsoperations?view=azure-python#query-by-pipeline-run-resource-group-name--factory-name--run-id--filter-parameters--custom-headers-none--raw-false----operation-config-

The below is to get the duration time of azure datafactory activity:

https://learn.microsoft.com/en-us/python/api/azure-mgmt-datafactory/azure.mgmt.datafactory.models.activityrun?view=azure-python#variables

(There is a variable named duration_in_ms, you can use this to get the duration time of the activity run.)

This is use Python to monitor pipeline:

https://learn.microsoft.com/en-us/azure/data-factory/monitor-programmatically#python

You can create a azure function app with a timetrigger to monitor the azure datafactory activity. This is the document of azure function timetrigger:

https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-timer?tabs=python

The basic idea is put the code that monitor the pipeline whether run more than N minutes in the logic body of azure function timetrigger. And then use the status of the azure function to reflect whether the pipeline running time of azure datafactory exceeds N hours.

Then use the alarm event of the azure function. The alarm events supported by azure for the azure function are as follows: (You can set an output binding of your azure function.)

In azure portal, you can find the alert in this place:

(Select Email/SMS message as the action type and give it your email address.)