async-http-client requests timing out with NettyAsyncHttpProvider

Question

I'm using async-http-client but I'm having trouble with NettyAsyncHttpProvider.

It gives this warning:

Oct 28, 2014 12:50:16 PM org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool
WARNING: Failed to get all worker threads ready within 10 second(s). Make sure to specify the executor which has more threads than the requested workerCount. If unsure, use Executors.newCachedThreadPool().

My problem here is even if I use a Executors.newCachedThreadPool() the following issue will persist.

Once the number of threads reaches the corePoolSize, requests get dropped. Whatever requestTimeoutInMs or other options I try, only a fraction of the responses come back and the threads in the pool stall.

ExecutorService executor = new CustomThreadPoolExecutor(2, 2, 60,
            TimeUnit.SECONDS, new LinkedBlockingQueue<Runnable>(), new ThreadFactory() {
        @Override
        public Thread newThread(Runnable r) {
            int number = threadCreationCount.incrementAndGet();
            LOG.info("newThread - {} ", number);

            Thread th = new Thread(r);
            th.setName("AsyncHttpClient-" + number);
            LOG.info("thread ={}", th);
            return th;
        }
    });

    AsyncHttpClientConfig.Builder builder = new AsyncHttpClientConfig.Builder();
    builder.setMaximumConnectionsTotal(-1)
            .setMaximumConnectionsPerHost(-1)
            .setConnectionTimeoutInMs(1000)
            .setIdleConnectionInPoolTimeoutInMs(60000)
            .setIdleConnectionTimeoutInMs(60000)
            .setRequestTimeoutInMs(3000)
            .setFollowRedirects(true)
            .setMaximumNumberOfRedirects(5)
            .setAllowPoolingConnection(true)
            .setIOThreadMultiplier(4)
            .build();

    builder.setExecutorService(executor);

    AsyncHttpClientConfig config = builder.build();
    AsyncHttpClient client = new AsyncHttpClient(new NettyAsyncHttpProvider(config), config);

    //Spin up 500 async requests to google.com
    for (int i = 1; i <= 500; i++) {
        LOG.info("i = {}", i);
        ListenableFuture<Response> future = client.prepareGet("http://www.google.com").execute(
                new AsyncCompletionHandler<Response>() {
                    @Override public Response onCompleted(Response response) throws Exception {
                        LOG.info("Response = {}, count = {}", response.getStatusCode(),
                                responseCount.incrementAndGet());
                        return response;
                    }

                    @Override
                    public void onThrowable(Throwable t) {
                        LOG.error("on throwable ={}", t);
                    }

                });
    }

Now If I change from NettyAsyncHttpProvider to ApacheAsyncHttpProvider all the requests are performed.

I created a sample project on github to showcase the issue. async-http-client-debugging

score 1 · Answer 1 · answered Dec 16 '15 at 03:43

I was getting the same issue when using the default executor service. I didn't debug into why, but using this configuration below, I didn't get the 10 second delay and warning message.

AsyncHttpClientConfig.Builder builder = new AsyncHttpClientConfig.Builder();
    builder.setConnectTimeout(3000)
            .setExecutorService(new ThreadPoolExecutor(0, 20,
                    60L, TimeUnit.SECONDS,
                    new SynchronousQueue<>()))
            .setRequestTimeout(3000)
            .setReadTimeout(3000)
            .build();

score -1 · Answer 2 · answered Jan 06 '16 at 12:47

That's not a threading issue. There's 2 things in play here:

you're sending a huge bunch of requests at the same time, without any back pressure, so you're trying to open 500 concurrent connections at the same time. You'd better have a queue and a dedicated Executor in front of AHC so you can limit and control the number of concurrent flying requests.
you're hammering google.com. They don't let you do that of course, and block your connect attempts.

PS: You would have get a faster answer if you would have asked on the AHC google group instead.

async-http-client requests timing out with NettyAsyncHttpProvider

2 Answers2