Azure Durable Function error handling: Is there a way to identify which retry you are on?

Question

The Durable Functions documentation specifies the following pattern to set up automatic handling of retries when an exception is raised within an activity function:

public static async Task Run(DurableOrchestrationContext context)
{
    var retryOptions = new RetryOptions(
        firstRetryInterval: TimeSpan.FromSeconds(5),
        maxNumberOfAttempts: 3);

    await ctx.CallActivityWithRetryAsync("FlakyFunction", retryOptions, "ABC");

    // ...
}

However I can't see a way to check which retry you're up to within the activity function:

[FunctionName("FlakyFunction")]
public static string[] MyFlakyFunction(
    [ActivityTrigger] string id, 
    ILogger log)
{
    // Is there a built-in way to tell what retry attempt I'm up to here?
    var retry = ??

    DoFlakyStuffThatMayCauseException();
}

EDIT: I know it can probably be handled by mangling some sort of count into the RetryOptions.Handle delegate, but that's a horrible solution. It can be handled manually by maintaining an external state each time it's executed, but given that there's an internal count of retries I'm just wondering if there's any way to access that. Primary intended use is debugging and logging, but I can think of many other uses.

Marc · Accepted Answer · 2019-08-20T17:38:11.030

There does not seem to be a way to identify the retry. Activity functions are unaware of state and retries. When the CallActivityWithRetryAsync call is made the DurableOrchestrationContext calls the ScheduleWithRetry method of the OrchestrationContext class inside the DurableTask framework:

public virtual Task<T> ScheduleWithRetry<T>(string name, string version, RetryOptions retryOptions, params object[] parameters)
{
    Task<T> RetryCall() => ScheduleTask<T>(name, version, parameters);
    var retryInterceptor = new RetryInterceptor<T>(this, retryOptions, RetryCall);
    return retryInterceptor.Invoke();
}

There the Invoke method on the RetryInterceptor class is called and that does a foreach loop over the maximum number of retries. This class does not expose properties or methods to obtain the number of retries.

Another workaround to help with debugging could be logging statements inside the activity function. And when you're running it locally you can put in a breakpoint there to see how often it stops there. Note that there is already a feature request to handle better logging for retries. You could add your feedback there or raise a new issue if you feel that's more appropriate.

To be honest, I think it's good that an activity is unaware of state and retries. That should be the responsibility of the orchestrator. It would be useful however if you could get some statistics on retries to see if there is a trend in performance degradation.

Thanks for your answer, it confirmed my suspicions. And thanks for pointing me to that feature request, that would also go a long way towards solving my problems. I see your point about the activity being unaware of state, and I'd probably be ok with having a callback in the RetryOptions that passed in some sort of retry state or context each time a retry attempt was about to be made. — pcdev, Aug 20 '19 at 21:20

Azure Durable Function error handling: Is there a way to identify which retry you are on?

1 Answers1