3

I am trying to create a combined resilience strategy for my Rest clients and for this I have written the following :

private static Policy circuitBreakerPolicy = Policy
     .Handle<TimeoutException>()
     .CircuitBreakerAsync(3, TimeSpan.FromSeconds(2));

And I am getting the error

Cannot implicitly convert type Polly.CircuitBreaker.AsyncCircuitBreaker to Polly.Policy

What am I missing ? I have checked a number of references online but I cannot come across a concise enough explanation of what causes this issue.

Usage : Eventually I want to combine the CircuitBreaker policy above with a WaitandRetry policy but I cannot figure out how to extract the combined policy :

(This is working)

public static IAsyncPolicy<HttpResponseMessage> CreateResiliencePolicy()
{
    var timeoutPolicy = Policy.TimeoutAsync(TimeSpan.FromSeconds(180));
 
    var waitAndRetryPolicy = Polly.Policy
        .Handle<HttpRequestException>()
        .OrResult<HttpResponseMessage>(r => r.StatusCode == HttpStatusCode.InternalServerError)
        .WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(3, retryAttempt)),
        (result, timeSpan, context) =>
        {
        });
    
    return timeoutPolicy.WrapAsync(waitAndRetryPolicy);
}

This is what I want (Not working) :

public static IAsyncPolicy<HttpResponseMessage> CreateResiliencePolicy()
{
    var circuitBreakerPolicy = Policy
        .Handle<TimeoutException>()
        .CircuitBreakerAsync(3, TimeSpan.FromSeconds(2));

    var waitAndRetryPolicy = Polly.Policy
        .Handle<HttpRequestException>()
        .OrResult<HttpResponseMessage>(r => r.StatusCode == HttpStatusCode.InternalServerError)
        .WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(3, retryAttempt)),
        (result, timeSpan, context) =>
        {
        });
   
    return circuitBreakerPolicy.WrapAsync(waitAndRetryPolicy);

     //I want to do this instead
     //public static Policy resilientAsyncStrategy =circuitBreakerPolicy.WrapAsync(waitAndRetryPolicy);

     //Then return resilientAsyncStrategy
     //return resilientAsyncStrategy;
}

And then use the returned policy instance as :

public async Task<IEnumerable<PaymentDetailDto>> GetAsync()
{
    var items = await resilientAsyncStrategy.ExecuteAsync(async () => await client.GetAsync());
    return items;
}
Peter Csala
  • 17,736
  • 16
  • 35
  • 75
Golide
  • 835
  • 3
  • 13
  • 36

1 Answers1

2

Polly defines the following four abstract Policy types:

Method Function
Sync Policy Policy<TResult>
Async AsyncPolicy AsyncPolicy<TResult>

So, in your case the circuitBreakerPolicy should be defined like this:

private static AsyncPolicy circuitBreakerPolicy = Policy
     .Handle<TimeoutException>()
     .CircuitBreakerAsync(3, TimeSpan.FromSeconds(2));

Whenever you want to combine / chain two (or more) policies then you should consider to use PolicyWrap (reference). Please bear in mind that policy chain works in an escalation way which means if the inner policy can't handle the problem then it will propagate that to the next outer policy.

Please also bear in mind that policies should be compatible with each other. So, if one of them is async then other one should be as well. If the inner returns something then the outer should do the same.

So, your CreateResiliencePolicy could look like this:

public static AsyncPolicy<HttpResponseMessage> CreateResilienceStrategy()
{
    var circuitBreakerPolicy = Policy<HttpResponseMessage>
        .Handle<TimeoutException>()
        .CircuitBreakerAsync(3, TimeSpan.FromSeconds(2));

    var waitAndRetryPolicy = Polly.Policy
        .Handle<HttpRequestException>()
        .OrResult<HttpResponseMessage>(r => r.StatusCode == HttpStatusCode.InternalServerError)
        .WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(3, retryAttempt)),
        (result, timeSpan, context) =>
        {
        });

    return Policy.WrapAsync(circuitBreakerPolicy, waitAndRetryPolicy);
}

Please also bear in mind that ordering matters:

  1. Policy.WrapAsync(circuitBreakerPolicy, waitAndRetryPolicy) you have an inner Retry and an outer CB
  2. Policy.WrapAsync(waitAndRetryPolicy, circuitBreakerPolicy) you have an inner CB and an outer Retry

The following two lines are equivalent:

circuitBreakerPolicy.WrapAsync(waitAndRetryPolicy);
Policy.WrapAsync(circuitBreakerPolicy, waitAndRetryPolicy)

If you want to change the ordering of the policies then your strategy will work in a different way. If you want to use the CB as inner policy and the retry as outer then you should amend the waitAndRetryPolicy to handle BrokenCircuitException as well.

Couple of months ago I have put together a sample application which demonstrates how can you incrementally design your resilience strategy.


UPDATE: Add sample code

Testing the retry logic

I've used the following console app to test your retry policy:

private static HttpClient client = new HttpClient();
public static async Task Main(string[] args)
{
    var strategy = CreateResilienceStrategy();
    await strategy.ExecuteAsync(async (ct) =>
        await client.GetAsync("https://httpstat.us/500", ct)
        , CancellationToken.None);
    Console.WriteLine("Finished");
}

public static AsyncPolicy<HttpResponseMessage> CreateResilienceStrategy()
{
    var circuitBreakerPolicy = Policy<HttpResponseMessage>
        .Handle<TimeoutException>()
        .CircuitBreakerAsync(3, TimeSpan.FromSeconds(2),
            onBreak: (_, __) => Console.WriteLine("Break"),
            onReset: () => Console.WriteLine("Reset"),
            onHalfOpen: () => Console.WriteLine("HalfOpen"));

    var waitAndRetryPolicy = Polly.Policy
        .Handle<HttpRequestException>()
        .OrResult<HttpResponseMessage>(r => r.StatusCode == HttpStatusCode.InternalServerError)
        .WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(3, retryAttempt)),
        onRetryAsync: (_, ts, ___) =>
        {
            Console.WriteLine($"Retry, penalty: {ts.Seconds} secs");
            return Task.CompletedTask;
        });

    return Policy.WrapAsync(circuitBreakerPolicy, waitAndRetryPolicy);
}

The output:

Retry, penalty: 3 secs
Retry, penalty: 9 secs
Retry, penalty: 27 secs
Finished

As you can see,

  • it had performed all retries without luck
  • it did not trigger the circuit breaker
  • strategy.ExecuteAsync returned a response where the status code is 500
  • it continued the work after the Http call

Testing the circuit breaker logic

Because the CB policy was setup to trigger for TimeoutException that's why I set the client.Timeout to 1 millisecond in the Main. With that the application crashes with TaskCanceledException.

It does that because HttpClient throws TaskCanceledException in case of Timeout instead of TimeoutException. If you would use polly's Timeout policy then you would receive TimeoutRejectedException.

So, here is my amended to code to be able to test CB

private static HttpClient client = new HttpClient();
public static async Task Main(string[] args)
{
    client.Timeout = TimeSpan.FromMilliseconds(1);
    var strategy = CreateResilienceStrategy();
    try
    {
        await strategy.ExecuteAsync(async (ct) =>
            await client.GetAsync("https://httpstat.us/500", ct)
            , CancellationToken.None);
        Console.WriteLine("Finished");
    }
    catch (Exception ex)
    {
        Console.WriteLine("Failed with " + ex.GetType().Name);
    }    
}

public static AsyncPolicy<HttpResponseMessage> CreateResilienceStrategy()
{
    var circuitBreakerPolicy = Policy<HttpResponseMessage>
        .Handle<OperationCanceledException>()
        .Or<TimeoutRejectedException>()
        .CircuitBreakerAsync(1, TimeSpan.FromSeconds(2),
            onBreak: (_, __) => Console.WriteLine("Break"),
            onReset: () => Console.WriteLine("Reset"),
            onHalfOpen: () => Console.WriteLine("HalfOpen"));

    var waitAndRetryPolicy = Polly.Policy
        .Handle<HttpRequestException>()
        .Or<OperationCanceledException>()
        .OrResult<HttpResponseMessage>(r => r.StatusCode == HttpStatusCode.InternalServerError)
        .WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(3, retryAttempt)),
        onRetryAsync: (_, ts, ___) =>
        {
            Console.WriteLine($"Retry, penalty: {ts.Seconds} secs");
            return Task.CompletedTask;
        });

    return Policy.WrapAsync(waitAndRetryPolicy, circuitBreakerPolicy);
}

Modifications:

  • Set the timeout to 1 milliseconds
  • Changed the consecutive failed count of CB from 3 to 1
  • Changed the chaining: retry outer, cb inner
  • Added OperationCanceledException conditions for both policies

The output

Break
Retry, penalty: 3 secs
HalfOpen
Break
Retry, penalty: 9 secs
HalfOpen
Break
Retry, penalty: 27 secs
HalfOpen
Break
Failed with OperationCanceledException
Peter Csala
  • 17,736
  • 16
  • 35
  • 75
  • The line " Policy.WrapAsync(circuitBreakerPolicy, waitAndRetryPolicy); " shows exception Polly.Retry.AsyncRetryPolicy to Polly.IAsyncPolicy – Golide Jul 11 '21 at 17:34
  • @Golide I've extended my post with 2 sample applications which shows that the provided code works. If you still experience some error or exception please let me know. Please also try to share as many details regarding that as possible. – Peter Csala Jul 12 '21 at 07:12
  • @Golide Did the suggested solution work for you? Do you need any further information? – Peter Csala Jul 13 '21 at 08:08
  • 1
    I have implemented the console app , works perfectly. I will update (within the day) once im done testing the same in the WebAPI ... – Golide Jul 13 '21 at 10:42
  • @Golide Have you had a chance to test it in a WebAPI project? – Peter Csala Jul 16 '21 at 13:46
  • I have an error "no overload for method 'GetAsync' takes 1 argument" . I am using RestEase with HttpClient as follows : public PaymentClient(IConfiguration configuration, IDiscoveryClient discoveryClient) { var handler = new DiscoveryHttpClientHandler(discoveryClient); var httpClient = new HttpClient(handler, false) { BaseAddress = new Uri(configuration.GetValue("PricingServiceUri")) }; client = RestClient.For(httpClient); } – Golide Jul 16 '21 at 15:13
  • public async Task> GetAsync() { var strategy = ResilienceStrategy(); try { var items = await strategy.ExecuteAsync(async (ct) => await client.GetAsync("www.htttp.com", ct) , CancellationToken.None); Console.WriteLine("Finished"); } catch (Exception ex) { } return items; } – Golide Jul 16 '21 at 15:26
  • @Golide Which GetAsync shows this error? In the provided code you have a couple of them... – Peter Csala Jul 16 '21 at 22:45
  • await client.GetAsync("www.htttp.com", ct) , CancellationToken.None); .This is the instance of RestClient ... taken from client = RestClient.For(httpClient); – Golide Jul 17 '21 at 00:12
  • @Golide I'm not familiar with RestEase. But based on the provided error message, I suspect it is a compilation error. The message says there is no overload which anticipates a single parameter. On the other hand your provided code shows that you are calling with two. Is my understanding correct? – Peter Csala Jul 17 '21 at 09:19
  • I have reverted to this form which seems to be working (not errors). Confirm the resilienceStrategy is being applied correctly ? public async Task> GetAsync() { var items = await strategy.ExecuteAsync(async () => (HttpResponseMessage)await client.GetAsync()); return (IEnumerable)items; } – Golide Jul 17 '21 at 17:58
  • @Golide The CancellationToken is only needed of you want to apply Timeout policy as well. Otherwise you can omit it, just like you did. – Peter Csala Jul 17 '21 at 19:12
  • @Golide I want to emphasize that casting HttpResponseMessage directly to IEnumerable feels wrong. Most probably you need to deserialize the response body to retieve the received data. But I think this is out of scope for this question. – Peter Csala Jul 17 '21 at 19:15