How to add dynamic retry policies using dotnet core and Polly

Question

I have a dotnet core (2.1) Console application and I am using Polly to wrap a segment of my code with a retry policy. This works fine with a simple use case shown below:

private void ProcessRun() 
{
    var policy = Policy.Handle<SocketException>().WaitAndRetryAsync(
                 retryCount: 3
                 sleepDurationProvider: attempt => TimeSpan.FromSeconds(10),
                 onRetry: (exception, calculatedWaitDuration) => 
                 {
                    Console.WriteLine($"Retry policy executed for type SocketException");
                 });

    try
    {
        CancellationTokenSource _cts = new CancellationTokenSource()

        PollyRetryWaitPolicy.ExecuteAsync(async token => {
           MyOperation(token);
        }, _cts.Token)
        .ContinueWith(p => {
           if (p.IsFaulted || p.Status == TaskStatus.Canceled)
           {
                Console.WriteLine("faulted or was cancelled");
           }
        })
        .ConfigureAwait(false);
    }
    catch (Exception ex) {
     Console.WriteLine($"Exception has occurred: {ex.Message}");
    }
}

I then test it using this code:

private void MyOperation() 
{
    Thread.Sleep(2000);
    throw new SocketException();
}

When the code is executed, the SocketException is caught as expected.

I was looking a my flexible way to wrap my code with multiple policies instead of one. I changed the code to dynamically add a number of wrapped Polly Retry Policies together, to allow for more than one error type to be caught and easily change the exceptions I'm looking out for. I changed the code to this:

internal PolicyWrap PollyRetryWaitPolicy;

public void AddRetryWaitPolicy<T>(int waitTimeInSeconds, int retryAttempts)
    where T: Exception
{

    // Setup the polly policy that will be added to the executing code.
    var policy = Policy.Handle<T>().WaitAndRetryAsync(
                retryCount: retryAttempts, 
                sleepDurationProvider: attempt => TimeSpan.FromSeconds(waitTimeInSeconds), 
                onRetry: (exception, calculatedWaitDuration) => 
                {
                    Console.WriteLine($"Retry policy executed for type {typeof(T).Name}");
                });

    if (host.PollyRetryWaitPolicy == null)
    {
        // NOTE: Only add this timeout policy as it seems to need at least one
        // policy before it can wrap! (suppose that makes sense).
        var timeoutPolicy = Policy
            .TimeoutAsync(TimeSpan.FromSeconds(waitTimeInSeconds), TimeoutStrategy.Pessimistic);
        PollyRetryWaitPolicy = policy.WrapAsync(timeoutPolicy);
    }
    else
    {
        PollyRetryWaitPolicy.WrapAsync(policy);
    }
}

private void ProcessRun() 
{
    AddRetryWaitPolicy<SocketException>(10, 5);
    AddRetryWaitPolicy<InvalidOperationException>(5, 2);

    try
    {
        Console.WriteLine($"Calling HostedProcess.Run() method. AwaitResult: {awaitResult}");

        CancellationTokenSource _cts = new CancellationTokenSource()

        PollyRetryWaitPolicy.ExecuteAsync(async token => {
           MyOperation(token);
        }, _cts.Token)
        .ContinueWith(p => {
           if (p.IsFaulted || p.Status == TaskStatus.Canceled)
           {
                Console.WriteLine("Process has faulted or was cancelled");
           }
        })
        .ConfigureAwait(false);
    }
    catch (Exception ex) {
     Console.WriteLine($"Exception has occurred: {ex.Message}");
    }
}

When I test with this code, the above code works as expected and is retried 5 times.

private void MyOperation() 
{
    Thread.Sleep(2000);
    throw new SocketException();
}

BUT when I try with the following, it does not retry 2 times as expected (it does not retry at all):

private void MyOperation() 
{
    Thread.Sleep(2000);
    throw new InvalidOperationException();
}

What am I doing wrong? The point of all of the above is to dynamically multiple policies as I require. Is there a better way to do with other than WrapPolicy?

score 1 · Accepted Answer · answered Oct 02 '18 at 09:52

Here:

if (host.PollyRetryWaitPolicy == null)
{
    // NOTE: Only add this timeout policy as it seems to need at least one
    // policy before it can wrap! (suppose that makes sense).
    var timeoutPolicy = Policy
        .TimeoutAsync(TimeSpan.FromSeconds(waitTimeInSeconds), TimeoutStrategy.Pessimistic);
    PollyRetryWaitPolicy = policy.WrapAsync(timeoutPolicy);
}
else
{
    PollyRetryWaitPolicy.WrapAsync(policy);
}

it appears the if branch and the else branch take an inconsistent approach to sequencing the retry policy and the timeout policy in the wrap.

In the if branch, the timeout policy is wrapped inside the retry policy. So the timeout acts as a timeout-per-try.
In the else branch, any existing retry-and-timeout policy is wrapped outside the new retry policy. So the timeout policy is outside the new retry policy.
- The new retry policy is set to retry after waitTimeInSeconds; but the timeout policy is also set to timeout the execution after waitTimeInSeconds. So as soon as the first retry (for a second-or-subsequent-configured retry policy) occurs, the timeout aborts the whole execution. So the retry never happens, as you observed.

To cure this you might change the else branch to:

PollyRetryWaitPolicy = policy.WrapAsync(PollyRetryWaitPolicy);

Background: See the PolicyWrap wiki recommendations on ordering policies in a PolicyWrap. According to whether you place TimeoutPolicy inside or outside a RetryPolicy, TimeoutPolicy acts (when inside) as timeout-per-try, or (when outside) as overall timeout across all tries.

How to add dynamic retry policies using dotnet core and Polly

1 Answers1