using Apache's AsyncHttpClient in a storm bolt

Question

I have a bolt that is making an API call (HTTP Get) for every tuple. to avoid the need to wait for the response, I was looking to use the apache HttpAsyncClient.

after instantiating the client in the bolt's prepare method, the execute method constructs the URL from the tuple and calls sendAsyncGetRequest(url):

private void sendAsyncGetRequest(String url){

    httpclient.execute(new HttpGet(url), new FutureCallback<HttpResponse>() {

        @Override
        public void completed(HttpResponse response) {
            LOG.info("Response Code : " + response.getStatusLine());
            LOG.debug(response.toString());
        }

        @Override
        public void failed(Exception ex) {
            LOG.warn("Async http request failed!", ex);
        }

        @Override
        public void cancelled() {
            LOG.warn("Async http request canceled!");
        }
    });
}

the topology deploys but the Storm UI shows an error:

java.lang.RuntimeException: java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:12

I feel that there is not enough data to check this problem. How httpClient is instanciated? Can you post the full error stack? — zenbeni, Aug 26 '14 at 08:43

score 0 · Answer 1 · answered Dec 15 '14 at 02:42

I got this to work with no issues. the key things to note are:

declare the client on the bolt class scope

public class MyRichBolt extends BaseRichBolt {
    private CloseableHttpAsyncClient httpclient;

Instantiate and stat the client in the bolt's prepare method

@Override
public final void prepare(Map stormConf, TopologyContext context, OutputCollector collector) {
    try {
        // start the http client
        httpclient = HttpAsyncClients.createDefault();
        httpclient.start();
        // other initialization code ... 
    } catch (Throwable exception) {
    // handle errors
    }
}

make the calls in the bolt's execute method

@Override
public final void execute(Tuple tuple) {
    // format the request url
    String url = ... 
    sendAsyncGetRequest(url);
}


private void sendAsyncGetRequest(String url){
    logger.debug("Async call to URL...");
    HttpGet request = new HttpGet(url);
    HttpAsyncRequestProducer producer = HttpAsyncMethods.create(request);
    AsyncCharConsumer<HttpResponse> consumer = new AsyncCharConsumer<HttpResponse>() {

        HttpResponse response;

        @Override
        protected void onResponseReceived(final HttpResponse response) {
        this.response = response;
        }

        @Override
        protected void onCharReceived(final CharBuffer buf, final IOControl ioctrl) throws IOException {
            // Do something useful
        }

        @Override
        protected void releaseResources() {
        }

        @Override
        protected HttpResponse buildResult(final HttpContext context) {
            return this.response;
        }
    };

    httpclient.execute(producer, consumer, new FutureCallback<HttpResponse>() {

        @Override
        public void completed(HttpResponse response) {
            // do something useful with the response
            logger.debug(response.toString());
        }

        @Override
        public void failed(Exception ex) {
            logger.warn("!!! Async http request failed!", ex);
        }

        @Override
        public void cancelled() {
            logger.warn("Async http request canceled!");
        }
    });
}

An explanation of the root cause would be more helpful. I.e., why does your prescribed solution work? — Mark, Feb 23 '17 at 18:45

score 0 · Answer 2 · answered Jun 26 '16 at 13:25

Are you shutting down the client (client.close();) in your main flow before the callback can execute?

The error is saying that the IO path has already been closed. In general, instances of async clients should be re-used for repeated requests and destroyed only when "ALL" requests have been made, e.g. at application shutdown.

using Apache's AsyncHttpClient in a storm bolt

2 Answers2

Linked