how to scale azure durable funciton

Question

so i am using Durable functions for a 2 tasks.

1. I download files form sftp and upload them to blob. and i keep on adding name of these files in a string list.

2. then i pass this list to another function that has to perform calculations on these files.

    [FunctionName("MainOrch")]
    public async Task<List<string>> RunOrchestrator(
        [OrchestrationTrigger] IDurableOrchestrationContext context)
    {
        var fileUploaded = new List<string>();
   
        // Replace "hello" with the name of your Durable Activity Function.
        fileUploaded = await context.CallActivityAsync<List<string>>("SFTPDownloadAndUpload", null);
        foreach (var fileName in fileUploaded)
        {
            await context.CallActivityAsync("PBARParsing", fileName);
        }
        return fileUploaded;
    }
   

and the parsing function that does the calculations is setup like this

  [FunctionName("PBARParsing")]
    public async Task PBARParsing([ActivityTrigger] string name,
        [Blob("pbar-staging/{name}", FileAccess.Read, Connection = "pbarBlobConnectionVault")] Stream myBlob,
        ILogger log)
    {
        try
        {
            log.LogInformation("**********Starting" + name);

my question is that is it going to scale the parsing function i mean if 10 files are given to this function is it going to run 10 instances of parsing function one for each file or do i have to do something else to do that ?

Answer 1

In your current code, the parsing activity will run in series, one by one. This is because you await the activity Task inside the loop. If you want to run all of them in parallel (fan-out fan-in pattern), you need to collect the tasks and await them all at once:

 [FunctionName("MainOrch")]
 public async Task<List<string>> RunOrchestrator(
     [OrchestrationTrigger] IDurableOrchestrationContext context)
 {
     var fileUploaded = new List<string>();

     // Replace "hello" with the name of your Durable Activity Function.
     fileUploaded = await context.CallActivityAsync<List<string>>("SFTPDownloadAndUpload", null);

     var parseTasks = new List<Task>(fileUploaded.Count);
     foreach (var fileName in fileUploaded)
     {
         var parseTask = context.CallActivityAsync("PBARParsing", fileName);
         parseTasks.Add(parseTask);
     }

     await Task.WhenAll(parseTasks);

     return fileUploaded;
 }