FallbackAsync Method stops REST calls

Question

public static IAsyncPolicy<HttpResponseMessage> GetSaveExceptionPolicy(
            IServiceProvider serviceProvider)
{

    return Policy
        .HandleResult<HttpResponseMessage>(x => x.StatusCode == HttpStatusCode.Forbidden)
        .FallbackAsync(new HttpResponseMessage(HttpStatusCode.OK),
            d =>
            {
                var mediator = serviceProvider.GetService<IMediatorService>();
                mediator.SaveFailedOnInvalidLock();
                return Task.CompletedTask;
            });
}

 serviceCollection
    .AddHttpClient<IMyClient, MyClient>(p =>
    {
        p.BaseAddress = baseAddress;
        p.Timeout = TimeSpan.FromMinutes(10);
        
    })
    .ConfigurePrimaryHttpMessageHandler(sp => new HttpClientHandler() { MaxConnectionsPerServer = 10})
    .AddPolicyHandler((serviceProvider, request) => GetSaveExceptionPolicy(serviceProvider));

this is the mediator call

public void SaveFailedOnInvalidLock()
{
    _logger.Error($"Data Point Saving failed");
    SaveFailedOnInvalidLock?.Invoke();
}

This MyClient Post methods are called very frequently. For example at a given second, there are 7 calls going on. and for each second there are 7 of them continues.

The application works fine until the Post method returns a "HttpStatusCode.Forbidden". When it returns a Forbidden, some of the next calls do not reach the REST end point.

Is this due to some connections not being available? eg: few connections are lost due to "fobidden" status and polly fallback policy? (because max limit is 10) ?

or is this due to Async and its timing such that the connections are still in use when the 2nd lot of calls come in?

thanks in Advance

Do you know when and why does the remote server return with 403? Is the remote server protected with authentication? — Peter Csala, Mar 29 '21 at 08:31
403 is the expected return code . The REST end point is returning it to indicate a specific error. and we are broadcasting that to our system via "mediator.SaveFailedOnInvalidLock();" — mmek123, Mar 29 '21 at 10:34
it is expected that once a 403 happens, all calls to "MyClient" will return 403 . For me, all the calls to "MyClient" do not get processed — mmek123, Mar 29 '21 at 10:35
I'm not familiar with the mediator API. Is `SaveFailedOnInvalidLock` sync or async? — Peter Csala, Mar 29 '21 at 10:49
Please try to provide all the necessary information to have a minimal, [reproducible example](https://stackoverflow.com/help/minimal-reproducible-example). Please amend your question. — Peter Csala, Mar 29 '21 at 11:55

score 0 · Answer 1 · edited Mar 31 '21 at 07:06

I was able to solve this by returning the same http response message in the fallback action ( rather than creating a new HTTPResponseMessage)

I guess since i have MaxConnectionsPerServer = 10 and I'm using 7 of them at a given moment, if I return new HttpResponseMessage(HttpStatusCode.OK) there are some handlers that are not disposed or stuck.

This is my new policy

public static AsyncFallbackPolicy<HttpResponseMessage> GetDataPointSaveExceptionPolicy(
    IServiceProvider serviceProvider)
{
   return Policy 
       .HandleResult<HttpResponseMessage>(x=>
           x.StatusCode ==  HttpStatusCode.Forbidden)
       .FallbackAsync(
           (res, cx, ct) =>
           {
               var dataServiceMediator = serviceProvider.GetService<IDataServiceMediatorService>();
               dataServiceMediator.DataPointSaveFailedOnInvalidLock();

               return Task.FromResult(res.Result); 
           },
           (ct, cx) => Task.CompletedTask);
}

This returns Task.FromResult(res.Result) and there are no dispose issues and the calls to the MyClient continues to Reach REST end point.

FallbackAsync Method stops REST calls

1 Answers1