Can't execute circuit breaker after containerization

Question

I have a .NET 5 application, divided in microservices. I implemented the Circuit Breaker pattern using the Microsoft library and Polly.

Everything was tested and working accordingly - locally. But when I try to run with docker, if none of the microservices is down, it works perfectly, all the requests have responses (the expected ones).

On the contrary, when I put one of the microservices down and try to test and see if the circuit is open (getting the response with that information) it simply times out, returning an exception in which a task was not completed (timeout).

How can I solve this? I'm using the http port to run docker, I have tried to disable the httpsRedirection in Startup.cs as well as making the requests either with http and https, but none of these latter was successful. I am really out of ideas here. Below is an example of a microservice(with the relevant code of the question) and the respective docker file:

OrchAuth.cs:

services.AddControllers();
services.AddHttpClient<ISearchCommunicationServiceWatchables, SearchRESTCommunicationServiceWatchables>("Watchables")
     .SetHandlerLifetime(TimeSpan.FromMinutes(1))
     .AddPolicyHandler(GetRetryPolicy())
     .AddPolicyHandler(GetCircuitBreakerPolicy());
services.AddHttpClient<IUsersCommunicationService, UsersRESTCommunicationService>("Users")
     .SetHandlerLifetime(TimeSpan.FromMinutes(1))
     .AddPolicyHandler(GetRetryPolicy())
     .AddPolicyHandler(GetCircuitBreakerPolicy());
services.AddHttpClient<IUserPreferencesService, UserPreferencesService>("UserPreferences")
     .SetHandlerLifetime(TimeSpan.FromMinutes(1))
     .AddPolicyHandler(GetRetryPolicy())
     .AddPolicyHandler(GetCircuitBreakerPolicy());
services.AddHttpClient<ISearchCommunicationServiceBooks, SearchRESTComunicationServiceBooks>("Books")
     .SetHandlerLifetime(TimeSpan.FromMinutes(1))
     .AddPolicyHandler(GetRetryPolicy())
     .AddPolicyHandler(GetCircuitBreakerPolicy());
...

static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy()
{
    Random jitterer = new ();
    return HttpPolicyExtensions
        .HandleTransientHttpError()
        .WaitAndRetryAsync(2, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt))  // exponential back-off: 2, 4, 8 etc
            + TimeSpan.FromMilliseconds(jitterer.Next(0, 1000))); // plus some jitter: up to 1 second);
}

static IAsyncPolicy<HttpResponseMessage> GetCircuitBreakerPolicy()
{
    return HttpPolicyExtensions
        .HandleTransientHttpError()
        .CircuitBreakerAsync(2, TimeSpan.FromSeconds(10));
}

The docker file:

FROM mcr.microsoft.com/dotnet/aspnet:5.0 AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443

FROM mcr.microsoft.com/dotnet/sdk:5.0 AS build
WORKDIR /src
COPY ["boomoseries-OrchAuth-api/boomoseries-OrchAuth-api.csproj", "boomoseries-OrchAuth-api/"]
RUN dotnet restore "boomoseries-OrchAuth-api/boomoseries-OrchAuth-api.csproj"
COPY . .
WORKDIR "/src/boomoseries-OrchAuth-api"
RUN dotnet build "boomoseries-OrchAuth-api.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "boomoseries-OrchAuth-api.csproj" -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENV USERS_HOST=http://host.docker.internal:5020/Users
ENV PREFS_HOST=http://host.docker.internal:5024/UserPreferences/Favorites
ENV SEARCH_HOST=http://host.docker.internal:5018/api/v1/Search
ENTRYPOINT ["dotnet", "boomoseries-OrchAuth-api.dll"]

Could you please describe what is the expected behaviour from the resilience policies? You have a retry and circuit breaker but it seems like they are unaware of each other. — Peter Csala, May 30 '22 at 08:01
The expected behavior should be similar to the one I'm having locally: I basically try two times before opening the circuit, and the wait and retry increment exponentially with the addition of a jitterer. The circuit breaker then (after two unsuccessful tries, and being it configured to handle some HTTP responses (codes) with the method HandleTrasientHttpError) will open and for 10 seconds and it won't be able to process requests. I believe the problem is the lack of timeout configuration in the http client, I will try to configure this and will provide feedback. — Miguel Ferreira, May 30 '22 at 13:59
The error response I got recently was related to the default value of timeout: 100 seconds! And locally I always have the "backup" of the host actively refusing the connection, hence opening the circuit. Perhaps in docker this works differently — Miguel Ferreira, May 30 '22 at 14:01
Are you sure that you want to have this strategy?: Initial attempt fails, CB is still closed so 1. retry attempt issued, It fails, CB is now open, 2. retry attempt is issued, It fails with `BrokenCircuitException`. Would it make more sense to make your retry logic CB aware? — Peter Csala, May 31 '22 at 08:57

score 3 · Accepted Answer · answered May 31 '22 at 08:26

The problem has been solved. The issue was the following: the circuit breaker policy is configured to handle HTTP errors of type 5XX and 408, and I was not getting any of these as a response (when running docker): I was getting "blank". This does not happen locally because of the host actively refusing connection, being able to trigger and open the circuit. So for this, I had to configure the timeout policy in Polly, throw the exception and let the circuit breaker policy handle it (code below).

 public static IAsyncPolicy<HttpResponseMessage> GetCircuitBreakerPolicy()
    {
        return HttpPolicyExtensions
            .HandleTransientHttpError()
            .Or<TimeoutRejectedException>()
            .CircuitBreakerAsync(2, TimeSpan.FromSeconds(10));
    }

Another detail was the order in which this was configured in the startup: the circuit breaker policy had to be implemented BEFORE the retry policy (code below).

public static IHttpClientBuilder ConfigHttpClient<TInterface, TClass>(this IServiceCollection services, string httpClientName)
        where TInterface : class
        where TClass : class, TInterface
    {

       return services.AddHttpClient<TInterface, TClass>(httpClientName)
           .AddPolicyHandler(Startup.GetCircuitBreakerPolicy())
           .AddPolicyHandler(Startup.GetRetryPolicy())
           .AddPolicyHandler(Policy.TimeoutAsync<HttpResponseMessage>(3));

    }

Can't execute circuit breaker after containerization

OrchAuth.cs:

1 Answers1