Problem
Even though it seems quite appealing to have a registry with a full of policies unfortunately it is error-prone. At the first glace it seems that it provides great flexibility by allowing to the consumer to be able to combine whatever policies he/she wants.
But there is a problem, which is related to the escalation. If the inner policy fails then Polly will escalate the problem to the next outer policy (in the policy chain) which may or may not aware of the inner policy.
Example #1
Let's stick with your example where you have a Retry and a Circuit Breaker policies.
The consumer can register them in the following two orders:
- Circuit Breaker >> Retry
- Retry >> Circuit Breaker
- In the former case the retry will not throw any Polly specific exception.
- But in the latter case Polly might throw
BrokenCircuitException.
- So depending on the use case your retry policy might need to be aware of that.
Example #2
Let's suppose you have a Timeout policy as well. This is where things can get quite complicated.
The Timeout can be used either as a local or a global timeout:
- Retry >> Circuit Breaker >> Timeout
- Timeout >> Retry >> Circuit Breaker
+1. Timeout >> Retry >> Circuit Breaker >> Timeout
- In the first case your Timeout should not be aware of the any other Polly policies, but Circuit Breaker and Retry might need to be aware of the
TimeoutRejectedException.
- In the second case your Timeout might need to be aware of
BrokenCircuitException...
- In the bonus case how can you define a single policy which could be reused for local and global timeouts as well?
Design
Fortunately Polly can help us with Wrap. With this you can explicitly combine two or more policies in a given order. This helps you define strategies not just individual policies.
So instead of exposing policies you can expose strategies which combines a set of policies to achieve the desired behaviour.
Let's stick with the Retry >> Circuit Breaker >> Timeout example:
- Timeout aware of: -
- Circuit breaker aware of:
HttpRequestExcetion, TimeotRejectedException
- Retry aware of:
HttpRequestExcetion, TimeotRejectedException, BrokenCircuitException
Solution
Let's suppose we have the following abstraction to specify the parameters of the above three policies:
public class ResilienceSettings
{
public string BaseAddress { get; set; }
public int HttpRequestTimeoutInMilliseconds { get; set; }
public int HttpRequestRetrySleepDurationInMilliseconds { get; set; }
public int HttpRequestRetryCount { get; set; }
public int HttpRequestCircuitBreakerFailCountInCloseState { get; set; }
public int HttpRequestCircuitBreakerDelayInMillisecondsBetweenOpenAndHalfOpenStates { get; set; }
}
Now, let's see the policies:
private static IAsyncPolicy<HttpResponseMessage> TimeoutPolicy(ResilienceSettings settings)
=> Policy
.TimeoutAsync<HttpResponseMessage>( //Catches TaskCanceledException and throws instead TimeoutRejectedException
timeout: TimeSpan.FromMilliseconds(settings.HttpRequestTimeoutInMilliseconds));
private static IAsyncPolicy<HttpResponseMessage> CircuitBreakerPolicy(ResilienceSettings settings)
=> HttpPolicyExtensions
.HandleTransientHttpError() //Catches HttpRequestException or checks the status code: 5xx or 408
.Or<TimeoutRejectedException>() //Catches TimeoutRejectedException, which can be thrown by an inner TimeoutPolicy
.CircuitBreakerAsync( //Monitors consecutive failures
handledEventsAllowedBeforeBreaking: settings
.HttpRequestCircuitBreakerFailCountInCloseState, //After this amount of consecutive failures it will break
durationOfBreak: TimeSpan.FromMilliseconds(settings
.HttpRequestCircuitBreakerDelayInMillisecondsBetweenOpenAndHalfOpenStates)); //After this amount of delay it will give it a try
private static IAsyncPolicy<HttpResponseMessage> RetryPolicy(ResilienceSettings settings)
=> HttpPolicyExtensions
.HandleTransientHttpError() //Catches HttpRequestException or checks the status code: 5xx or 408
.Or<BrokenCircuitException>() //Catches BrokenCircuitException, so whenever the broker is open then it refuses new requests
.Or<TimeoutRejectedException>() //Catches TimeoutRejectedException, which can be thrown by an inner TimeoutPolicy
.WaitAndRetryAsync( //Monitors the above anomalies
retryCount: settings.HttpRequestRetryCount, //After (this amount + 1) attempts it gives up
sleepDurationProvider: _ =>
TimeSpan.FromMilliseconds(settings.HttpRequestRetrySleepDurationInMilliseconds)); //After a failed attempt it delays the next try with this amount of time
And finally the extension method for registration:
public static class ResilientHttpClientRegister
{
public static IServiceCollection AddXYZResilientStrategyToHttpClientProxy<TInterface, TImplementation>
(this IServiceCollection services, ResilienceSettings settings)
where TInterface: class
where TImplementation: class, TInterface
{
var (serviceUri, combinedPolicy) = CreateParametersForXYZStrategy<TInterface>(settings);
services.AddHttpClient<TInterface, TImplementation>(
client => { client.BaseAddress = serviceUri; })
.AddPolicyHandler(combinedPolicy); //Retry > Circuit Breaker > Timeout (outer > inner)
return services;
}
private static (Uri, IAsyncPolicy<HttpResponseMessage>) CreateParametersForXYZStrategy<TInterface>(ResilienceSettings settings)
{
Uri serviceUri = Uri.TryCreate(settings.BaseAddress, UriKind.Absolute, out serviceUri)
? serviceUri
: throw new UriFormatException(
$"Invalid url was set for the '{typeof(TInterface).Name}' resilient http client. " +
$"Its value was '{HttpUtility.UrlEncode(settings.BaseAddress)}'");
var combinedPolicy = Policy.WrapAsync(RetryPolicy(settings), CircuitBreakerPolicy(settings), TimeoutPolicy(settings));
return (serviceUri, combinedPolicy);
}
}
