Polly defines the following four abstract Policy types:
|
Method |
Function |
Sync |
Policy |
Policy<TResult> |
Async |
AsyncPolicy |
AsyncPolicy<TResult> |
So, in your case the circuitBreakerPolicy
should be defined like this:
private static AsyncPolicy circuitBreakerPolicy = Policy
.Handle<TimeoutException>()
.CircuitBreakerAsync(3, TimeSpan.FromSeconds(2));
Whenever you want to combine / chain two (or more) policies then you should consider to use PolicyWrap
(reference). Please bear in mind that policy chain works in an escalation way which means if the inner policy can't handle the problem then it will propagate that to the next outer policy.
Please also bear in mind that policies should be compatible with each other. So, if one of them is async then other one should be as well. If the inner returns something then the outer should do the same.
So, your CreateResiliencePolicy
could look like this:
public static AsyncPolicy<HttpResponseMessage> CreateResilienceStrategy()
{
var circuitBreakerPolicy = Policy<HttpResponseMessage>
.Handle<TimeoutException>()
.CircuitBreakerAsync(3, TimeSpan.FromSeconds(2));
var waitAndRetryPolicy = Polly.Policy
.Handle<HttpRequestException>()
.OrResult<HttpResponseMessage>(r => r.StatusCode == HttpStatusCode.InternalServerError)
.WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(3, retryAttempt)),
(result, timeSpan, context) =>
{
});
return Policy.WrapAsync(circuitBreakerPolicy, waitAndRetryPolicy);
}
Please also bear in mind that ordering matters:
Policy.WrapAsync(circuitBreakerPolicy, waitAndRetryPolicy)
you have an inner Retry and an outer CB
Policy.WrapAsync(waitAndRetryPolicy, circuitBreakerPolicy)
you have an inner CB and an outer Retry
The following two lines are equivalent:
circuitBreakerPolicy.WrapAsync(waitAndRetryPolicy);
Policy.WrapAsync(circuitBreakerPolicy, waitAndRetryPolicy)
If you want to change the ordering of the policies then your strategy will work in a different way. If you want to use the CB as inner policy and the retry as outer then you should amend the waitAndRetryPolicy
to handle BrokenCircuitException
as well.
Couple of months ago I have put together a sample application which demonstrates how can you incrementally design your resilience strategy.
UPDATE: Add sample code
Testing the retry logic
I've used the following console app to test your retry policy:
private static HttpClient client = new HttpClient();
public static async Task Main(string[] args)
{
var strategy = CreateResilienceStrategy();
await strategy.ExecuteAsync(async (ct) =>
await client.GetAsync("https://httpstat.us/500", ct)
, CancellationToken.None);
Console.WriteLine("Finished");
}
public static AsyncPolicy<HttpResponseMessage> CreateResilienceStrategy()
{
var circuitBreakerPolicy = Policy<HttpResponseMessage>
.Handle<TimeoutException>()
.CircuitBreakerAsync(3, TimeSpan.FromSeconds(2),
onBreak: (_, __) => Console.WriteLine("Break"),
onReset: () => Console.WriteLine("Reset"),
onHalfOpen: () => Console.WriteLine("HalfOpen"));
var waitAndRetryPolicy = Polly.Policy
.Handle<HttpRequestException>()
.OrResult<HttpResponseMessage>(r => r.StatusCode == HttpStatusCode.InternalServerError)
.WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(3, retryAttempt)),
onRetryAsync: (_, ts, ___) =>
{
Console.WriteLine($"Retry, penalty: {ts.Seconds} secs");
return Task.CompletedTask;
});
return Policy.WrapAsync(circuitBreakerPolicy, waitAndRetryPolicy);
}
The output:
Retry, penalty: 3 secs
Retry, penalty: 9 secs
Retry, penalty: 27 secs
Finished
As you can see,
- it had performed all retries without luck
- it did not trigger the circuit breaker
strategy.ExecuteAsync
returned a response where the status code is 500
- it continued the work after the Http call
Testing the circuit breaker logic
Because the CB policy was setup to trigger for TimeoutException
that's why I set the client.Timeout
to 1 millisecond in the Main
. With that the application crashes with TaskCanceledException
.
It does that because HttpClient throws TaskCanceledException
in case of Timeout instead of TimeoutException
. If you would use polly's Timeout policy then you would receive TimeoutRejectedException
.
So, here is my amended to code to be able to test CB
private static HttpClient client = new HttpClient();
public static async Task Main(string[] args)
{
client.Timeout = TimeSpan.FromMilliseconds(1);
var strategy = CreateResilienceStrategy();
try
{
await strategy.ExecuteAsync(async (ct) =>
await client.GetAsync("https://httpstat.us/500", ct)
, CancellationToken.None);
Console.WriteLine("Finished");
}
catch (Exception ex)
{
Console.WriteLine("Failed with " + ex.GetType().Name);
}
}
public static AsyncPolicy<HttpResponseMessage> CreateResilienceStrategy()
{
var circuitBreakerPolicy = Policy<HttpResponseMessage>
.Handle<OperationCanceledException>()
.Or<TimeoutRejectedException>()
.CircuitBreakerAsync(1, TimeSpan.FromSeconds(2),
onBreak: (_, __) => Console.WriteLine("Break"),
onReset: () => Console.WriteLine("Reset"),
onHalfOpen: () => Console.WriteLine("HalfOpen"));
var waitAndRetryPolicy = Polly.Policy
.Handle<HttpRequestException>()
.Or<OperationCanceledException>()
.OrResult<HttpResponseMessage>(r => r.StatusCode == HttpStatusCode.InternalServerError)
.WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(3, retryAttempt)),
onRetryAsync: (_, ts, ___) =>
{
Console.WriteLine($"Retry, penalty: {ts.Seconds} secs");
return Task.CompletedTask;
});
return Policy.WrapAsync(waitAndRetryPolicy, circuitBreakerPolicy);
}
Modifications:
- Set the timeout to 1 milliseconds
- Changed the consecutive failed count of CB from 3 to 1
- Changed the chaining: retry outer, cb inner
- Added
OperationCanceledException
conditions for both policies
The output
Break
Retry, penalty: 3 secs
HalfOpen
Break
Retry, penalty: 9 secs
HalfOpen
Break
Retry, penalty: 27 secs
HalfOpen
Break
Failed with OperationCanceledException