4

I need to upload WEB API response files into Azure Data Lake. Then I have to dump those files into Azure SQL tables.

Above both processes must be scheduled to execute on hourly basis. Should I use Azure Web Jobs or Azure Function.

AnshuK
  • 55
  • 1
  • 2

2 Answers2

3

Azure Data Factory is probably the better mechanism to drive this recurring hourly pipeline. More details here.

https://learn.microsoft.com/en-us/azure/data-factory/data-factory-scheduling-and-execution

Amit Kulkarni
  • 910
  • 4
  • 11
2

If you are running Azure Functions on Consumption plan, the function call must complete within 5 minutes, which might be not enough for big data sets.

For the rest Functions and Web Jobs are similar for your scenario. Functions are actually running on top of Web Jobs. And if you don't need any advanced features of Functions (e.g. bindings), I would go for a Job.

Mikhail Shilkov
  • 34,128
  • 3
  • 68
  • 107
  • How about using Data Factory for Extracting the Web API Data and copy to Data Lake Store..? – AnshuK Mar 16 '17 at 11:50