Downloading Multiple Files Parallelly or Asynchronously in Java

Question

Here I am trying to download multiple files one after another:

Environment - Java 1.6

public List<Attachment> download(List<Attachment> attachments)
{
  for(Attachment attachment : attachments) {
    attachment.setDownStatus("Failed");
    String destLocation = "C:\Users\attachments";
    try {
        String attUrl = attachment.getUrl();
        String fileName = attachment.getFileName();            
        URL url = new URL(attUrl);
        File fileLocation = new File(destLoc, fileName);
        FileUtils.copyURLToFile(url, fileLocation);
        if(fileLocation.exists()) {
           attachment.setDownStatus("Completed");
         }
       } catch(Exception e) {
          attachment.setDownStatus("Failed");
       } finally {
          attachment.setDestLocation(destLocation);
       }
   }
  return attachments;
}

I am downloading the file from provided URL (http://cdn.octafinance.com/wp-content/uploads/2015/07/google-hummingbird.jpg).

FileUtils.copyURLToFile(url, fileLocation);

The above code does its downloading job perfectly, without any issues.

My Problem:
If the list of attachments are more it will take more time, so I would like to make it an asynchronous or parallel process instead of downloading sequentially.

score 6 · Answer 1 · edited Aug 20 '19 at 19:06

6

Use Java 8 Streams in combination with ForkJoinPool

public List<Attachment> download(List<Attachment> attachments) throws InterruptedException, ExecutionException {

    ForkJoinPool forkJoinPool = new ForkJoinPool(attachments.size());

    return forkJoinPool.submit(() -> processAttachments(attachments)).get();
}

private List<Attachment> processAttachments(List<Attachment> attachments) {
    return attachments.stream().parallel().map(attachment -> processSingleAttachment(attachment)).collect(Collectors.toList());
}

private Attachment processSingleAttachment(Attachment attachment){
     //business logic to download single attachment
    .
    .
}

edited Aug 20 '19 at 19:06

0ddlyoko

310
4
14

answered Oct 12 '15 at 08:16

Vishal

2,711
7
33
42

Vishal, My project is based on Java 1.6, it would be good if you could able to provide solution based on java 1.6 – Uppicharla Oct 12 '15 at 08:20
1

Oh , I didn't see the Java version mentioned earlier. In that case the solution by Boris below looks good to me. – Vishal Oct 12 '15 at 08:23

Boris Pavlović · Answer 2 · 2015-10-12T07:28:37.683

3

public List<Attachment> download(List<Attachment> attachments)
{
  ExecutorService executorService = Executors.newCachedThreadPool();
  for(final Attachment attachment : attachments){
    executorService.submit(new Runnable() {

        @Override
        public void run() {
          try{
            String attUrl = attachment.getUrl();
            String fileName = attachment.getFileName();
            String destLocation = "C:\Users\attachments";
            URL url = new URL(attUrl);
            String fileLocation = new File(destLoc, fileName);
            FileUtils.copyURLToFile(url, fileLocation);
            if(fileLocation.exists()) {
              attachment.setDownStatus("Completed");
            }
          }
          catch(Exception e){
            attachment.setDownStatus("Failed");
          }
        }
    });
 }
 executorService.shutdown();
 return attachments;
}

edited Oct 12 '15 at 07:28

answered Oct 12 '15 at 07:23

Boris Pavlović

63,078
28
122
148

This is an I/O-bound task. The number of available processors is a bad choice (or at least, not a good choice) for the pool size. Instead, use the number of files you want to download at once. – user253751 Oct 12 '15 at 07:27
Also, you might want to wait for all attachments to finish before returning. Also, you probably want to shut down the thread pool when done (or reuse the same thread pool for every call). – user253751 Oct 12 '15 at 07:28
Each attachment has a `downStatus` which can be used to monitor downloaded files. – Boris Pavlović Oct 12 '15 at 07:30
@Boris, the suggested code runs faster than my code, But this code doesn't guarantees about downStatus, some times for some attachments i am getting 'null'. is there any way to get out of this issue. – Uppicharla Oct 12 '15 at 10:14
1

@Uppicharla your original code doesn't guarantee that `downStatus` is not `null`. So why do you expect that this code makes such a guarantee? – Olivier Grégoire Oct 12 '15 at 10:51
@Olivier, Actually i posted some part of my actual code, Now i updated my post, please take a look once. – Uppicharla Oct 12 '15 at 11:18
I applied the Boris approach to my code, after that i observed some times 'downStatus' is coming as 'null'. – Uppicharla Oct 12 '15 at 11:28
you have to wait until it got initialized – Boris Pavlović Oct 12 '15 at 11:33
@Boris, One more observation, `attachment.setDestLocation(destLocation);` is also setting null in some cases. – Uppicharla Oct 12 '15 at 11:44

score 1 · Accepted Answer · answered Oct 12 '15 at 14:02

Actually, after carefully looking, Boris' code is faulty and will indeed not set some stuff sometimes. Here's a better version that fixes that:

public List<Attachment> download(List<Attachment> attachments) {
  ExecutorService executorService = Executors.newCachedThreadPool();
  List<Future<Attachment>> futures = new ArrayList<Future<Attachment>>();
  for (final Attachment attachment : attachments) {
    futures.add(executorService.submit(new Callable<Attachment>() {
      @Override
      public Attachment call() throws Exception {
        return doDownload(attachment);
      }
    }));
  }
  for (Future<Attachment> future: futures) {
    try {
      future.get();
    } catch (Exception ex) {
      // Do something
    }
  }
  return attachments;
}

private Attachment doDownload(Attachment attachment) throws Exception {
  attachment.setDownStatus("Failed");
  attachment.setDestLocation("C:\\Users\\attachments");
  String attUrl = attachment.getUrl();
  String fileName = attachment.getFileName();
  URL url = new URL(attUrl);
  File fileLocation = new File(attachment.getDestLocation(), fileName);
  FileUtils.copyURLToFile(url, fileLocation);
  if (fileLocation.exists()) {
    attachment.setDownStatus("Completed");
  }
  return attachment;
}

However, this is absolutely not optimal given your structure of Attachment and how you use it. I did not fix that: I only answered the question as it was asked.

Downloading Multiple Files Parallelly or Asynchronously in Java

3 Answers3

Linked