0

I need to find the best way to preform long running tasks on Azure. Scenario:

  1. User picks the dataset and filters on the web app (Azure App Service)
  2. Based on the requirements we create SQL query
  3. Query is executed against one or more Azure SQL databases
  4. Data is being exported in .csv format and uploaded to Azure Blob Storage

The main issues are that execution of some SQL queries can last for 2+ hours and resultset can have 100M+ rows.

I believe that Azure Functions (and subsequently Durable Functions) are not a option because of the timeout and memory usage.

  • Can you explain a bit more about your end to end scenario you want to enable? Why do you need the CSV files? What size SQL are you using? What happens to the CSV files after you export all of them? etc. – Conor Cunningham MSFT Jun 23 '21 at 22:16
  • Hi, Azure SQL databases in questions are GP: Gen5 16vCore and DTU S9. We need to provide a way for users to manually select the data from multiple databases, combine and filter them. After the export is complete users download the .csv files and import them to their system. In the future, blob pickup will be automated. – lijevosmetalo Jun 24 '21 at 11:02
  • sounds to me like you should use elastic queries on SQL DB or use SQL Server + multiple databases. – Conor Cunningham MSFT Jun 25 '21 at 13:03
  • You can set up a Kubernetes cluster and run Jobs. This can be cost effective too. – Chayan Bansal Jun 28 '21 at 16:39

1 Answers1

1

I believe that Azure Functions (and subsequently Durable Functions) are not a option because of the timeout and memory usage.

Timeout limit is only in consumption plan. If you want to get rid of timeout limit and want more memory, you can use premium plan or app service plan. And because azure function can scale out horizontally, you can split the result to multiple and then input the azure function.

And also, you can use web jobs.

Cindy Pau
  • 13,085
  • 1
  • 15
  • 27
  • Hi, thank you for your answer. The more I investigate, this workload seems less and less suitable for Azure, at least from cost perspective. – lijevosmetalo Jun 24 '21 at 11:04
  • @lijevosmetalo You can consider what to use according to your own budget. This is a [cost calculation tool](https://azure.microsoft.com/en-us/pricing/calculator/?service=functions#functionse9e50b69-a40b-49cf-acac-83786868417a). – Cindy Pau Jun 24 '21 at 11:16