Running a large amount of activity functions from data from a database

Question

We have a database with around 400k elements we need to compute. Below is shown a sample of an orchestrator function.

[FunctionName("Crawl")]
public static async Task<List<string>> RunOrchestrator(
[OrchestrationTrigger] DurableOrchestrationContext context)
{
   if (!context.IsReplaying)
   {
   }

   WriteLine("In orchistration");
   var outputs = new List<string>();

   var tasks = new Task<string>[3];

   var retryOptions = new RetryOptions(
       firstRetryInterval: TimeSpan.FromSeconds(60),
       maxNumberOfAttempts: 3);

   // Replace "hello" with the name of your Durable Activity Function.
   tasks[0] = context.CallActivityWithRetryAsync<string>("Crawl_Hello",retryOptions, "Tokyo");
   tasks[1] = context.CallActivityWithRetryAsync<string>("Crawl_Hello", retryOptions, "Seattle");
   tasks[2] = context.CallActivityWithRetryAsync<string>("Crawl_Hello",retryOptions, "London");

   await Task.WhenAll(tasks);

   return outputs;
}

Every time an activty is called the orchestration function is called. But I dont want to get 400k items from the database each time an activity is getting called. Would just just add all the activity code inside the if statement or what is the right approach here? I can't see that working with the WaitAll function.

Does this answer your question? [Is durable functions suited for high amount of activities?](https://stackoverflow.com/questions/59249027/is-durable-functions-suited-for-high-amount-of-activities) — Ian Kemp, Aug 13 '21 at 12:53

score 0 · Accepted Answer · answered Dec 18 '19 at 06:27

Looks like you've figured out the approach for this as you've mentioned in your other query but elaborating the same here for the benefit of others.

Ideally, you should have an activity function to first fetch all the data that you need first, batch them up and call another activity function that processes that data.

Since you have a large number of elements to compute on, its best to split compute into separate sub-orchestrators because the fan-in operation is performed on a single instance.

For further reading, there are some documented performance targets that could help when deploying durable functions.

Running a large amount of activity functions from data from a database

1 Answers1