-2

I have several processes represented by azure functions and I need to chain them. They are mainly interacting with the DB and working with the huge amount of data, so each function could take even 1 hour - therefor I can't use HTTP triggers and ADF pipelines (as the timeout is 230s).

What approach would you recommend for such a scenario: To chain couple of long-running functions (3-5 functions).

I have already tried Durable functions. They are working fine, but I had to move all functions inside of one Function app, which I wanted to avoid.

I would like to ask about experiences with Event Hub or Event Grid, if this technology is suitable for such a scenario as well and if yes, what approach would you choose and why? Or any other options?

Thanks a lot for your opinions.

Filip
  • 117
  • 2
  • 8

2 Answers2

0

Azure Batch may be a better fit for long-running jobs.

https://learn.microsoft.com/en-us/azure/batch/batch-technical-overview

For example, you can use an Azure Function app to create a batch job as in the sample architecture below.

enter image description here

Serkant Karaca
  • 1,896
  • 9
  • 8
-1

I would like to ask about experiences with Event Hub or Event Grid, if this technology is suitable for such a scenario as well and if yes, what approach would you choose and why? Or any other options?

Event Hub should be used as your entry point for Big Data. (when developing IOT solutions, you should use Azure Iot Hub as it's bi-directional).

About Event Grid, it's just a repository for events where you can publish/subscribe to events.

I think you should stay with Durable Functions. As another alternative, you can move the processing part to Azure Container Instances and use Service Bus Queue / Topic to notify the process status.

Thiago Custodio
  • 17,332
  • 6
  • 45
  • 90