Ning's AsyncHttpClient builds Futures way too slow (in this case)

Question

In my code there is loop which builds the Futures, which is way to slow. It takes about hundreds of milliseconds to build just 1 Future. I'am just talking about setting up the Future object and not about getting the Reponse. Here's the code:

while(settings.getCrawlerQueue().size() < settings.getCrawlerQueueSize()) {
    Task task = taskQueue.poll();
    task = setFutureInTask(assignment, task);
}

And the setFutureInTask() code:

public Task setFutureInTask(String assignment, Task task) {
    task.setParserAssignment(assignment);
        switch(assignment){

        case "stuff":
            task.setFuture(asyncClientStandard
                        .prepareGet("http://"+task.getDomain())
                        .execute()
                    );
            break;
[...]

I tried different Java JDKs and versions as suggest in some stuff that I read (can't find it :/ ), but that doesn't help. Also tried JDKAsyncHttpProvider instead of the default NettyAsyncHttpProvider, which is way faster, but also provides very much Flase-Negatives (it seems to cancel very(!) often valid domains). What I have noticed is: I live in Germany and german .de-domains are noticeable faster than domains from orther countries like Spain.

Any wild guess is highly appreciated! :)

[EDIT]

Simply switching to another DNS server solved the problem.

I haven't personally tested out the Async HTTP client, but have you tried koush's `AndroidAsync`? [link](https://github.com/koush/AndroidAsync) — The Nomad, Jul 21 '14 at 20:54

score 1 · Accepted Answer · answered Jul 21 '14 at 20:48

1

I have experienced a similar (if not identical) issue while writing and experimenting with my HTTP load test plugin. The name resolution seems to take place synchronously while creating the future, and if your DNS caching is not working properly, you pay for the whole DNS lookup roundtrip each time.

As to how to fix this issue, it is very difficult to say. I would usually just switch to another network interface which wouldn't show this problem.

answered Jul 21 '14 at 20:48

Marko Topolnik

195,646
29
319
436

The "DNS lookup roundtrip" thing makes very much sense, there's a DNS problem currently at my ISP. What do you mean exactly by "switch to another network interface"? – Crayl Jul 21 '14 at 20:56
In my case I have the option to go through the corporate network or through WiFi with a direct DSL link. – Marko Topolnik Jul 22 '14 at 08:30
Simply switching DNS servers helped :D – Crayl Jul 22 '14 at 12:09

Ning's AsyncHttpClient builds Futures way too slow (in this case)

1 Answers1