0

I have used CopyOnWriteArrayList collection object which holds 1000 URLs. each URL indicates a file.

I want to use Multithread pooling mechanism to download those URL files parallel.

Tried using below code :

    CopyOnWriteArrayList<String> fileList = DataExtractor.getRefLinks();

    ExecutorService threadPool = Executors.newFixedThreadPool(4);
    CompletionService<String> pool = new ExecutorCompletionService<String>(
            threadPool);

    for (int i = 0; i < fileList.size() ; i++){
        pool.submit(new StringTask(fileList));
    }

This is hitting the same URL 4 times. Might have done something wrong. Could you please suggest where it went wrong ?

My requirement is to pick 4 URLs (threads) at a time and start downloading them parallel till all the URLs in the List finish downloading.

Thanks.

Molay
  • 1,154
  • 2
  • 19
  • 42

1 Answers1

4

I don't know what StringTask is, but you seem to be passing the full list of URLs to it. Make the appropriate changes to only submit a single URL from the list

pool.submit(new StringTask(fileList.get(i)));

(Or use an iterator over the fileList, whichever is more appropriate for a CopyOnWriteArrayList.)

for (String url : fileList){
    pool.submit(new StringTask(url));
}
Sotirios Delimanolis
  • 274,122
  • 60
  • 696
  • 724