Http4s Blaze Client Builder wait queue full failure

Question

We have a use case where for a single incoming request, a microservice has to make many(nearly 1000 in worst case) outgoing HTTP calls to other microservice to fetch GET some details. Our service is built using Scala, Http4s and Cats-Effect and is using http4s-blaze-client library for making outbound HTTP calls.

Currently in production we are seeing the failure org.http4s.client.WaitQueueFullFailure: Wait queue is full and org.http4s.client.PoolManager: Max wait queue limit of 1024 reached, not scheduling. Once the service goes into this state, its never recovering from it and we are completely blocked.

Below is the Blaze Client configuration we are using:

BlazeClientBuilder[F](global)
  .withMaxWaitQueueLimit(1024)
  .withRequestTimeout(20.seconds)
  .resource
  .map { client =>
      ResponseLogger(logHeaders = false, logBody = true)(
      RequestLogger(logHeaders = true, logBody = true, redactHeadersWhen = Middleware.SensitiveHeaders)(client)
    )
  }

Initially we were using the default setting of 256 for max wait queue limit but then decided to increase to 512 and then to 1024. Currently even the 1024 is not working.

I am not sure if this happens when the outbound HTTP request is slow or times out. There is a possibility that the API response is slow sometimes(but that will still return within 20seconds timeout that we set). But I do not have sufficient evidence to claim that it is the case here.

We are currently using the version http4s-blaze-client_2.13:0.21.0-M6.

I am not sure if increasing the wait queue size further would help. Is it possible to implement custom logic within the service to check the wait queue size and wait before submitting the request to the client? Please advise how to get around with this issue. Any help would be really appreciated.

You should ask in the **http4s** **discord** channel: https://discord.gg/UpEn2mSx - Anyways, I would guess the wait queue is where HTTP requests that haven't finished are kept, if each call to your service requires `1000` external calls then it means that you need a queue size of extremely large size... I would look for a way to backpressure all this so you don't run too many HTTP calls concurrently. — Luis Miguel Mejía Suárez, May 18 '22 at 15:20
@LuisMiguelMejíaSuárez its not always that I execute 1000 external calls, it is the case in the worst case scenario. Could you please elaborate on your idea of backpressuring ? — Madhu Reddy, May 18 '22 at 16:10
You may put the HTTP client behind an interface, then the implementation may have an internal `Queue` of calls to execute, when the `Queue` is full it will either block the calling fiber or let it know that the call was not registered; it all depends on how you want to manage the back pressure. — Luis Miguel Mejía Suárez, May 18 '22 at 16:37

score 1 · Answer 1 · answered May 19 '22 at 21:16

Well, according to the comments, maxWaitQueueLimit is simply the “maximum number of requests waiting for a connection at any specific time”. So what would be the point of checking the wait queue size and waiting if it's full? http4s is already doing the waiting for you. The main difference is that if you implement the waiting yourself (e. g. by using a Semaphore and acquiring a permit every time you perform an HTTP request), then there's no limit to how many requests you can be waiting for. And that means that when there's high load on your server, you'll run out of memory and crash. This is presumably what the maxWaitQueueLimit is supposed to prevent in the first place.

Now, when you perform a lot of requests, they all end up in the http4s wait queue at first, except for those that can find a connection. The default for maxTotalConnections is 10, so when you fire off 1000 requests, 990 will end up in the wait queue. If in that moment another request comes in that triggers more than 34 requests, you've already overflowed the wait queue. Increasing the maxWaitQueueLimit much further seems perfectly reasonable to me given your situation. Assuming you can't somehow reduce the number of required HTTP requests, that is.

Http4s Blaze Client Builder wait queue full failure

1 Answers1