Polly Retry with HTTP Client is exceeding the Number of Max Retries and continues Retry till HTTP Timeout Occurred

Question

I have defined some Typed API Client via Dependency Injection in Start up like below.

public void ConfigureHttpClients(IServiceCollection services)
{
   services.AddHttpClient<IMyHttpClient,MyHttpClient>()    
           .AddPolicyHandler(GetRetryPolicy(Configuration));
}

And Get Retry Policy is defined like this :

private static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy(IConfiguration configuration)
{
    int numberOfretries = Convert.ToInt32(configuration["PollyRetries"]); // Value is set to 3
    TimeSpan delay = TimeSpan.FromSeconds(Convert.ToInt32(configuration["PollyMaxDelaySeconds"])); // Delay is set to 3

    return HttpPolicyExtensions
        .HandleTransientHttpError()
        .WaitAndRetryAsync(
        numberOfretries,
        retryAttempt => delay,
        onRetry: (response, span, retryCount, context) =>
        {
            Log.Information("Attempt # {retryCount}", retryCount);                 
        });
}

Retry Count is set to 3 and Delay is set to 3 seconds.

MyHttpClient is defined like this :

public class MyHttpClient : IMyHttpClient
{
   public MyHttpClient (HttpClient httpClient, IMemoryCache cache)
   {
     _httpClient = httpClient,
     _cache = cache
   }
}

But from the below image you can see even though the max retry is set to 3 the retry count is getting reset after 3 attempts and continues to retry until timeout

WebServerLogs

The Packages I am using related to Polly are -

<PackageReference Include="Microsoft.Extensions.Http.Polly" />
<PackageReference Include="Polly.Extensions.Http" />

Why Polly is retrying even though max limit is reached?
What can I do to resolve this ?

I have tried changing the package versions. But it did not help. Also looked at the Microsoft Documentation for Polly Implementation - implement-http-call-retries-exponential-backoff-polly. The implementation is exactly same as I have mentioned before.

If I wrap my client call with policy execute method like this :

_resiliencyPolicy = HttpPolicyExtensions
    .HandleTransientHttpError()
    .WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(3));

await _resiliencyPolicy.ExecuteAsync(async () =>
{
   await _myHttpClient.PostAsync(payload).ConfigureAwait(false);
}).ConfigureAwait(false);  

Then it only retries for 3 times after initial call. But configuring the HttpClient directly in startup.cs with policy, it continues to retry

Answer 1

Update : I was able to figure out the root cause of the HttpClient retrying multiple times (exceeding the max retry). Actually in Startup.cs, these httpclient configuration was done inside a wrapper method called ConfigureHttpClient as I mentioned before. But this ConfigureHttpClient method was called two times from two different places in Startup.cs. That applied those policies on the same HttpClient multiple times.

Now not sure about if the policies were applied 2 times on the httpclient as it did not seem so, because the client was retrying until the timeout happens. So exactly not sure what is the consequence if the policies are added multiple times to the same client. Can anybody explain this ? But once I remove the extra call and the ConfigureHttpClient is being called only once, then it only retried for 3 times and it stopped.

Answer 2

If you look at the logs you can see the following callback logs:

21.27.21 Attempt # 1
21.27.25 Attempt # 2
21.27.28 Attempt # 3
21.27.32 Attempt # 1
21.27.36 Attempt # 1
21.27.39 Attempt # 2
21.27.42 Attempt # 3
21.27.46 Attempt # 2
21.27.49 Attempt # 1
21.27.53 Attempt # 2
21.27.56 Attempt # 3
21.28.00 Attempt # 3
21.28.03 Attempt # 1
21.28.07 Attempt # 2
21.28.10 Attempt # 3

At first glance it does not make sense why do you have two Attempt # 1 subsequent lines, or why do you have an Attempt # 2 right after an Attempt # 3.

After you have confirmed that you have registered the policy twice it all makes sense. You have an inner and outer retry loops. So, by adding some indentation it reveals what happens

  21.27.21 Attempt # 1
  21.27.25 Attempt # 2
  21.27.28 Attempt # 3
21.27.32 Attempt # 1
  21.27.36 Attempt # 1
  21.27.39 Attempt # 2
  21.27.42 Attempt # 3
21.27.46 Attempt # 2
  21.27.49 Attempt # 1
  21.27.53 Attempt # 2
  21.27.56 Attempt # 3
21.28.00 Attempt # 3
  21.28.03 Attempt # 1
  21.28.07 Attempt # 2
  21.28.10 Attempt # 3

So, a retry count 3 means at most 4 attempts. The initial attempt + 3 retries. That's why you see 4 sequences of the inner attempts.

It can break with a timeout since if you add up all attempts and delays it might exceed the default timeout of HttpClient. This could be easily verified if you decrease the retry count to 2 and/or the sleep duration to 2 seconds.