Polling x amount of URL's every second with Java

Question

Is it even possible to poll let's say 10,000 URL's every second? I do keep threading in mind with the following code:

for (int i = 0; i < 10000; i++) {
    Executors.newScheduledThreadPool(10).scheduleWithFixedDelay(new Runnable() {
        @Override public void run() {
            // Poll URL here
        }
    }, i, 1000, TimeUnit.MILLISECONDS);
}

I do open a connection to the URL this way (I do close it later on after reading):

HttpURLConnection  connection = (HttpURLConnection) new URL("https://stackoverflow.com").openConnection();
connection.connect();

Then, I read the data of the outputstream:

InputStream inputStream = connection.getInputStream();

Then, I read the inputstream and close the HttpURLConnection.

When polling 200 URL's every second, everything works just fine. When polling 400+ url's, everything chokes. Connecting to a url takes 20-30 seconds.

I do test it on Tomcat with Eclipse on my own computer (not a server). When the connections take long to make, I can't even open webpages in the browser. The script uses about 20-30 Mbps while my internet has a maximum of 90 Mbps on the computer.

If you have any suggestions I would really like to hear them and try them.

Answer 1

Requesting using HEAD might reduce the amount of data that is sent back, allowing you to use less bandwidth, speeding up the queries.