I need to find the best way to preform long running tasks on Azure. Scenario:
- User picks the dataset and filters on the web app (Azure App Service)
- Based on the requirements we create SQL query
- Query is executed against one or more Azure SQL databases
- Data is being exported in .csv format and uploaded to Azure Blob Storage
The main issues are that execution of some SQL queries can last for 2+ hours and resultset can have 100M+ rows.
I believe that Azure Functions (and subsequently Durable Functions) are not a option because of the timeout and memory usage.