Apache Camel 2.13.1 Ning async-http-client 1.7.19
I'm trying to understand if I can use the reply-request pattern with Camel and the Ning async-http-client so that the route blocks until receipt of the HTTP response. ("Why are you using the async client then if you want to block until receiving the response?" you might ask. Not totally my choice; but apparently it has been previously found to be the most reliable http client where I'm working and that's just what we now use.)
So I have a camel setup that looks like this:
<camel:camelContext id="camelContext">
<camel:package>com.lijit.blackbird</camel:package>
<camel:threadPoolProfile id="selectionPool" poolSize="100" maxPoolSize="512" maxQueueSize="-1" />
<camel:route id="delivery">
<camel:from uri="direct:start" />
<camel:to uri="direct:select" />
<camel:to uri="direct:resolve" />
<camel:to uri="direct:finalStep" />
</camel:route>
<camel:route id="select">
<camel:from uri="direct:select" />
<camel:multicast executorServiceRef="selectionPool">
<camel:to uri="selectorType1" />
<camel:to uri="selectorType2" />
...
<camel:to uri="selectorTypeN" />
</camel:multicast>
</camel:route>
<camel:route id="resolve">
<camel:from uri="direct:resolve" />
<camel:split stopOnException="false" parallelProcessing="true">
<camel:method bean="myResolutionSplitter" />
<camel:to uri="bean:contentFetchingAsyncHttpClient" />
</camel:split>
</camel:route>
</camel:camelContext>
The gist of what's occurring is that some set of "selectors" pick "stuff". Said stuff includes a URL from which some content can be fetched. The message is thus split into each of the URLs that requires "resolution" (or the fetching of content to say it more plainly).
The fetching of this content is done using this Ning async-http-client, the code looking basically like this:
public class ContentFetchingAsyncHttpClient extends AsyncCompletionHandler<Response> {
private Future<Response> future;
private final AsyncHttpClient httpClient;
...
@Handler
public void fetch(URL url) {
Request request = new RequestBuilder()
.setMethod("GET")
.setUrl(url.toExternalForm())
.build();
future = httpClient.executeRequest(request, this);
}
@Override
public Response onCompleted(Response response) { /* do stuff with fetched content */ }
}
So my problem is this: the fetch() method sends off the HTTP request(s) for each of the URLs the message was split into. Since fetch() returns before a response is received, once all the requests are made the Camel route continues to finalStep which actually needs the retrieved content...which hasn't actually come back yet.
I was thinking the request-reply pattern might help here, but the examples all seem to be JMS-oriented and I am not clear how I would convert to the bean/POJO world. I was imagining that the onCompleted() method would need somehow to "reply". Any concrete suggestions would be very welcomed!
Moreover, now I've typed this out I'm not so sure this is the right pattern anyway. What I really want to achieve is n-number of HTTP requests occurring concurrently, and for the route to only progress when all of them have completed. A request-reply on the http request/response seems like it would serialize them rather than allow them to run concurrently... so any suggestions on a better architectural approach would also be highly welcome.
