4

Is there a way to control the maximum degree of parallelism when implementing the fan out/fan in pattern on Azure Durable Functions?

I'm currently implementing this pattern to perform a data loading process but I'm hitting database limits on DTU's because of the number of operations is too high for the database to handle.

The solution I'm thinking about involves using the following properties:

  • maxConcurrentActivityFunctions
  • maxConcurrentOrchestratorFunctions

of the (host.json) file in conjuntion with:

The first 2 properties should limit the number of parallel function executions per host, and the WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT should limit the number of hosts.

Would this be a correct approach for limiting the max degree of parallelism?

Chris Gillum
  • 14,526
  • 5
  • 48
  • 61
sanfalero
  • 372
  • 2
  • 18

1 Answers1

6

The approach you describe is the best approach for limiting max parallelism, globally. Just be careful that WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT is not always reliable, at the current time of writing. Later in the year, the team plans to make this a fully reliable setting.

This may not apply to your case, but for the benefit of other readers who find this question I've added one more thing to consider. If you have a single orchestration, one simple C# technique you can use to limit max parallelism is to do something like this:

static async Task FanOutFanInAsync(
    string functionName,
    object[] workItems,
    int maxParallelism)
{
    var inFlight = new HashSet<Task>();
    foreach (var item in workItems)
    {
        if (inFlight.Count > maxParallelism)
        {
            Task finished = await Task.WhenAny(inFlight);
            inFlight.Remove(finished);
        }

        inFlight.Add(ctx.CallActivityAsync(functionName, item));
    }

    await Task.WhenAll(inFlight);
}

This will allow you to limit how many activity functions you fan out to at a single time for a single orchestration instance.

Chris Gillum
  • 14,526
  • 5
  • 48
  • 61