0

I would like to create a pipeline in Data Factory and I would like to use Azure Function for some C# code which will download files from some web services, etc. The problem is that only HTTP Triggered function is supported in Azure Factory and HTTP Triggered function has duration limited for 230 seconds. https://learn.microsoft.com/en-us/azure/azure-functions/functions-scale

Is there any workaround how to trigger azure function with longer duration from data factory? I need to do it synchronously, because another processes will work with downloaded data.

Thank you very much for any ideas.

smeidak
  • 285
  • 1
  • 3
  • 13

1 Answers1

1

The simplest approach would be to just deploy your function app in an App Service Plan instead.

But if you require true serverless experience of the consumption plan, you could try using Durable Functions along with a pipeline setup that polls the status endpoint returned (as shown here) by the durable functions orchestrator.

I believe you would have to use

  • Until
  • Web
  • Set Variable
  • Wait

Basically, you would be polling the status endpoint until the runtimeStatus is set to "Completed".

Depending on your use case, Azure Batch Service might be something you'd want to look into too which also has a connector to Azure Data Factory.

PramodValavala
  • 6,026
  • 1
  • 11
  • 30
  • Thanks a lot. This is very good answer, most of these solutions I have already tested. As you said, actually there is not any directly possible solution for Azure factory without looping and checking status of function/webjob or custom application called by Azure Batch – smeidak Mar 27 '19 at 20:47