Durable Functions: Get failed tasks in Fan out pattern

Question

I'm using the fan-out pattern to execute a list of tasks that have a possibility of failing. So I'm using the retry strategy to retry a couple times. Right now, if the SaveActivity fails after the last retry then it drops into the catch block and i loose the succeeded tasks.

Is there a way, I could filter out the tasks that have failed after the last try and then move on with the successful list?

Is it possible to get the retry attempt in the activity so that i could suppress the exception and flag the entity?

List<Task<AssetSyncResult>> tasks = modeList.
    Select(r => context.CallActivityWithRetryAsync<AssetSyncResult>("SaveActivity", new RetryOptions(TimeSpan.FromSeconds(20),3), r)).
    ToList();
try{
    AssetSyncResult[] syncResults = await Task.WhenAll(tasks);
} catch(Exception e){
    //rescue the workflow
}

Getting the results from the Task is a no go. After recovering from the exception thrown by Task.WhenAll the control never gets to filters that check the status of the Task. Here's how i've changed the code:

try
{
    AssetSyncResult[] syncResults = await Task.WhenAll(tasks);
}
catch (System.Exception)
{
    
}

List<AssetSyncResult> resultsPassed = tasks.Where(r => r.IsCompleted).Select(r => r.Result).ToList();
List<AssetSyncResult> resultsFailed = tasks.Where(r => r.IsFaulted).Select(r => r.Result).ToList();

If any of my requests fail after the last retry attempt then Task.WhenAll throws an exception so I wouldn’t be able to access any property on Task or the inner object. — webber, Jan 04 '19 at 02:44
Why can't you read the status of the `Task` objects in `tasks`? — mjwills, Jan 04 '19 at 02:55
@mjwills ive tried getting the status of the tasks after the exception but it doesnt work. I've appears as if the orchestration can't get past the Task.WhenAll when any of the tasks have failed. — webber, Jan 04 '19 at 11:46

score 1 · Answer 1 · answered Oct 29 '19 at 20:18

1

If you want to get the list of messages for the failed functions use:

List<AssetSyncResult> resultsFailed = tasks.Where(r => r.IsFaulted).Select(r => r.Exception.Message).ToList();

answered Oct 29 '19 at 20:18

loganville

11
1
2

score 0 · Answer 2 · answered Jan 04 '19 at 12:05

0

Finally I've got this working by swallowing the exception thrown by Task.WhenAll and then running a filter to get the tasks that have succeeded. In my previous attempt, i was trying to get the tasks that had failed, but i realized that it was an incorrect approach. We should get the succeeded tasks and then compute the ones that have failed.

Here's what my code looks like

try
{
    AssetSyncResult[] syncResults = await Task.WhenAll(tasks);
}
catch (System.Exception)
{

}

var passedResults = tasks.Where(r => r.Status == TaskStatus.RanToCompletion).Select(r => r.Result).ToList();

answered Jan 04 '19 at 12:05

webber

1,834
5
24
56

If your original code never reached `resultsPassed`, I am surprised that this solution works. _Great job solving it though!_ – mjwills Jan 04 '19 at 12:09
so the key here is that you can't access the Task's result object if the Task failed. Previously, when i was filtering on "r.IsCompleted" the failed task was also considered to be "IsComplete". So I changed the filter to TaskStatus.RanToCompletion instead to get only the succeeded tasks. Hope that makes sense – webber Jan 04 '19 at 12:22
But you said it never even _reached_ that line. _I know about `RanToCompletion` - that is why I hinted you towards that in my first comment._ – mjwills Jan 04 '19 at 12:22

score 0 · Answer 3 · answered Jun 01 '22 at 11:27

2 years late to the party, but the issue with the snippet below is that if Task.WhenAll throws early in the process, all tasks/activities may not have completed when execution gets to the filtering.

try
{
    AssetSyncResult[] syncResults = await Task.WhenAll(tasks);
}
catch (System.Exception)
{

}

List<AssetSyncResult> resultsPassed = tasks.Where(r => r.IsCompleted).Select(r => r.Result).ToList();
List<AssetSyncResult> resultsFailed = tasks.Where(r => r.IsFaulted).Select(r => r.Result).ToList();

I have implemented a kind of ActivityResultWrapper, and the context.CallActivity is done in a try-catch, which will make sure my result wrappers always have either the result or the exception.

Pseudocode:

class ActivityResultWrapper<T> 
{ 
    public bool Succeeded => Failure == null;
    public T Result {get; set;}
    public Exception Failure {get; set;}
}

async Task<ActivityResult<T>> CallActivity<T>(IDurableFunctionContext context, string activity)
{
    try 
    {
        return new ActivityResultWrapper(){ Result = context.CallActivity<T>(activity) };
    }
    catch(Exception e)
    {
        return new ActivityResultWrapper(){ Failure = e };
    }
}

This allows the Task.WhenAll in the Orchestrate method will only complete when all activities have either completed or failed.

Durable Functions: Get failed tasks in Fan out pattern

3 Answers3