Email errors from all activities in ADF at once

Question

In Azure Data Factory I created a pipeline with several copy data activities. As soon as one of them crashes I want that it continues to the next step and that it sends an e-mail with the error.

The mail error action is correct. When I link it to just one activity and let that one crash on purpose it does send the mail correctly. But when I link it to multiple and it crashes (even the last step) it doesn't send an e-mail.

What am I doing wrong here?

see image

The e-mail step is quite simple, it's set that it triggers an Logic App which sends the actual mail via office365.

Answer 1

In adf, the flows act as AND condition. So based on your above image, an email will be sent only when all activities tagged to the web activity will fail. So to achieve your scenario, you would have to use a combination of on skip, on completion and on failure flows. Reference blog: https://datasharkx.wordpress.com/2021/08/19/error-logging-and-the-art-of-avoiding-redundant-activities-in-azure-data-factory

Answer 2

As soon as one of them crashes I want that it continues to the next step and that it sends an e-mail with the error.

All the connections marked "on failure" are the problem. Although what you created makes logical. When an activity receives connections from several dependents, it will not begin to run until all of the dependencies have reported in. for this you can use pipeline dependency called on completion to run next activity even if previous activity is failed.

And another problem in your approach is if one activity is executed successfully and you are trying to fetch error from that in web activity msg It will throw an error

You have to give web activity to each activity to send mail if they fail and give On Completion to all activities It will run all activities irrespective previous activity is failing or not.

Or If your main intension is to get mail if activities failed in pipeline you could create alert on particular pipeline with all activities with Failed activity runs metrics It would send you mail if any activity failed in that particular pipeline