Polly does not retry even though an exception is caught

Question

I have the following configuration of my HTTP service:

services
   .AddHttpClient<ISomeHttpService, SomeHttpService>((Action<HttpClient>) (client =>
      {
        client.BaseAddress = new Uri(configuration.BaseAddress);
        client.Timeout = TimeSpan.FromSeconds(10);
      }))
  .AddPolicyHandler(GetPollyPolicy())
  .AddHttpMessageHandler<DistributedCachedHttpClientHandler>();

// .....


private static IAsyncPolicy<HttpResponseMessage> GetPollyPolicy()
{
    return HttpPolicyExtensions
        .HandleTransientHttpError()
        .OrInner<Exception>() // I will probably use TimeoutException, for now I'm checking all possible options
        .WaitAndRetryAsync(2, _ => TimeSpan.FromMilliseconds(100), 
        (exception, timeSpan) =>
        {
           _ = "breakpoint";
        });
}

It works really well, if the request made by SomeHttpService returns 500. Polly tries to retry 2 more times.

However, if the request takes more time than 10 seconds (which is more than the configured timeout), retries do not occur. I even set a breakpoint within the Polly's action and I see that the breakpoint is hit when the timeout occurs! However, when I continue the program, no retries are executed and my web API fails with 500 directly after.

What's wrong?

I also tried using Or<Exception>() instead of OrInner<Exception>().

score 3 · Accepted Answer · answered Dec 06 '22 at 14:38

TL;DR: The observed behaviour is the way how HttpClient's Timeout overarches all retry attempts.

The root cause of your problem is that HttpClient's Timeout is considered as global rather than local timeout:

Global means overarching, includes all the retry attempts
Local means per retry attempt

If you want to define a local timeout then you have to combine Retry and Timeout policies:

var timeoutPolicy = Policy.TimeoutAsync<HttpResponseMessage>(5);
var retryPolicy = HttpPolicyExtensions
    .HandleTransientHttpError()
    .OrInner<TimeoutRejectedException>()
    .WaitAndRetryAsync(2, _ => TimeSpan.FromMilliseconds(10));

var combinedPolicy = Policy.WrapAsync(retryPolicy, timeoutPolicy);

Please note that the order of the parameters of the Policy.WrapAsync matters.

The above example defines a per request timeout policy (inner) and a retry policy (outer), which is aware of the potential timeout (TimeoutRejectedException).

You can define two timeout policies, where one acts locally whereas the other acts globally:

var combinedPolicy = Policy.WrapAsync(globalTimeout, retryPolicy, localTimeout);

That's a cool answer, thanks! Do I understand correctly that your last example (with `globalTimeout`) should be used when I'd like to have some smaller local timeout and some greater global timeout which will cause the whole policy to stop when it's crossed? — mnj, Dec 06 '22 at 15:05
@mnj Either use HttpClient's Timeout with some greater timespan or define a global timeout policy. Both work fine, the difference is thrown exception: OperationCanceledException vs TimeoutRejectedException. — Peter Csala, Dec 06 '22 at 15:36

Polly does not retry even though an exception is caught

1 Answers1