0

I have a task to export large data to excel/csv. The data consist of approximately between 10 - 20 millions. Below is approach

Client ----> ServiceBus Queue -----> Blob storage -----> SignalR ------> Client.

From client I send a message to servicebus queue, then in queue listener get all the data, write to excel file, push the file to blob storage, send the blob storage link to client using signalr service and finally download the file from blob storage in client.

Since the process to convert sql data to excel is taking long time to run, I am little confused to choose from the existing serverless computing services like servicebus, event hubs and azure functions.

For the above scenario which is best to choose for long running task servicebus/event hubs/azure functions. Kindly help me to identify the right architecture to implement the above scenario.

Thanks in advance.

Purushothaman
  • 519
  • 4
  • 16
  • How long is a long time on average? – Peter Bons Sep 01 '20 at 06:55
  • azure function dont suit for long time running. – Cindy Pau Sep 01 '20 at 07:15
  • @PeterBons I have not monitored how long it takes to execute, but approximately it takes 3mins for 10 million data – Purushothaman Sep 01 '20 at 07:22
  • Just to clarify: Of those options that you named neither servicebus or event hubs are compute offerings. Only Azure Functions are. SB and EH are messaging services – silent Sep 01 '20 at 08:07
  • 1
    As long as your task is well under 10 minutes, Functions should be a good fit https://stackoverflow.com/a/47447550/1537195 – silent Sep 01 '20 at 08:09
  • @silent Thanks for the info, but I am trying to identify the best architectural approach using azure services to export data in excel or csv. I could use some messaging service if it fits my requirements. – Purushothaman Sep 01 '20 at 08:12
  • @silent the 10m time limit is for azure functions on the "consumption" plan. There is no upper execution on the Azure functions on a "dedicated" App Service Plan. Purushothaman, just use an Azure function on a dedicated service plan. Other options don't offer compute. And I don't think we're ready to introduce azure data factory... etc. – Troy Witthoeft Sep 01 '20 at 12:28
  • Can you give some more details regarding the flow? Somewhere you are talking about converting data from sql to csv, something [Azure Data Factory (ADF)](https://azure.microsoft.com/en-us/services/data-factory/) could do. The ADF pipeline could be triggered by you and when the csv file is ready a signal could be send to the client to tell a download is ready. – Peter Bons Sep 01 '20 at 20:51
  • By the way, by asking for options about long running compute tasks you ignore possible better (azure) solutions that can do the actual job. You are already focusing on the how with a narrow perspective. – Peter Bons Sep 01 '20 at 20:53

0 Answers0