-1

Once a day I want to download multiple CSV files (could be several hundred of them), parse them, do some transforming and write them to another file or database. I want to have an overview which processing failed and which succeeded and prefereably trigger a retry on those that failed. So I would need some kind of batch/job processing in Azure. Which services could be used for that? Is that possible with these services?

  • Storage queue with Azure functions
  • Azure batch with spot VMs?

What other services could I look into?

stefan.at.kotlin
  • 15,347
  • 38
  • 147
  • 270
  • Give LogicApps a go. Nice easy logging and retry functionality and look at using the Advanced Data Operations connector … https://learn.microsoft.com/en-au/connectors/advanceddataoperatio/ … Cheap and (likely) has everything you need to transform your CSV to other structures. – Skin May 14 '23 at 21:13
  • Thanks! What if I wouldn't have one download of e.g. 1000 files and then the processing, but instead e.g. 100 times (download 10 files and process them)? So in other words, repeat a logic app 100 times with different parameters/input. Is Logic app intended for that or is it more intended for end users? – stefan.at.kotlin May 14 '23 at 21:39
  • 1
    If it helps, LogicApps has parallel processing built in and as for targeting end users, there’s not much in Azure that’s designed for end users. PowerAutomate cloud is the citizen based tool, it runs off the LogicApps engine. LA is designed for enterprise integration scenarios. There are small gaps though, hence why I suggest looking into that third party connector. – Skin May 14 '23 at 22:05

1 Answers1

1

There are quite a few Azure Services that could perform this task. Selection would really come down to your existing skills and preference. These are the one's I'd start evaluating (but not an exhaustive list );

  1. Azure Data Factory - This service is designed for precisely your use case, ETL of files.

  2. Databricks - This is a "big data" service preferred by data professionals and will likely be overkill for your scenario.

  3. Logic Apps - A "low code" solution that can operate as you've described. I do worry that it'll end up being a bit clunky and costly for you, depending on the transformations you'll be doing. Often for more complex activities you'll end up needing to write code in Azure Functions and call those functions from Logic Apps.

GordonBy
  • 3,099
  • 6
  • 31
  • 53