Faster way of copying data in Java?

Question

I have been given a task of copying data from a server. I am using BufferedInputStream and output stream to copy the data and I am doing it byte by byte. Even though it is running but It is taking ages to copy the data as some of them are in 100's MBs, so definitely it is not gonna work. Can anyone suggest me any alternate of Byte by Byte copy so that my code can copy file that are in few Hundred MBs. Buffer is 2048.

Here is how my code look like:

static void copyFiles(SmbFile[] files, String parent) throws IOException {

  SmbFileInputStream input = null;
  FileOutputStream output = null;
  BufferedInputStream buf_input = null;
  try {
    for (SmbFile f : files) {
      System.out.println("Working on files :" + f.getName());
      if (f.isDirectory()) {

        File folderToBeCreated = new File(parent+f.getName());
        if (!folderToBeCreated.exists()) {
          folderToBeCreated.mkdir();
          System.out.println("Folder name " + parent
                + f.getName() + "has been created");
        } else {
          System.out.println("exists");

        }

        copyFiles(f.listFiles(), parent +  f.getName());
      } else {

        input = (SmbFileInputStream) f.getInputStream();

        buf_input = new BufferedInputStream(input, BUFFER);

        File t = new File(parent + f.getName());
        if (!t.exists()) {
          t.createNewFile();
        }
        output = new FileOutputStream(t);

        int c;

        int count;
        byte data[] = new byte[BUFFER];

        while ((count = buf_input.read(data, 0, BUFFER)) != -1) {
          output.write(data, 0, count);
        }
      }
    }
  } catch (IOException e) {
    e.printStackTrace();

  } finally {
    if (input != null) {
      input.close();
    }
    if (output != null) {
      output.close();
    }
  }
}

Now that you've shown the code, you're *not* reading byte by byte, unless BUFFER is 1. (Your exception handling could do with some work, by the way... that's almost never an appropriate catch block.) — Jon Skeet, Dec 06 '12 at 16:39
@JonSkeet you mean i should put try catch for each file so that even if one file fails rest should not be affected — Max, Dec 06 '12 at 16:42
@JonSkeet Can you suggest me what should I do regarding try,catch block many thanks — Max, Dec 06 '12 at 16:44
@Batman: Well what do you *want* to happen if there's an IO failure? Do you really want to keep going? You've declared that your method can throw IOException, so why catch it at all in your method? — Jon Skeet, Dec 06 '12 at 16:53
@JonSkeet actually that is incase some error occurs while closing the stream — Max, Dec 06 '12 at 16:55

score 16 · Accepted Answer · answered Dec 06 '12 at 16:39

16

Here is a link to an excellent post explaining how to use nio channels to make copies of streams. It introduces a helper method ChannelTools.fastChannelCopy that lets you copy streams like this:

final InputStream input = new FileInputStream(inputFile);
final OutputStream output = new FileOutputStream(outputFile);
final ReadableByteChannel inputChannel = Channels.newChannel(input);
final WriteableByteChannel outputChannel = Channels.newChannel(output);
ChannelTools.fastChannelCopy(inputChannel, outputChannel);
inputChannel.close();
outputChannel.close()

answered Dec 06 '12 at 16:39

Sergey Kalinichenko

714,442
84
1,110
1,523

@Batman It may, because `Channels.newChannel` takes `InputStream`, and `SmbFileInputStream` is an `InputStream`. I would definitely give it a try. – Sergey Kalinichenko Dec 06 '12 at 17:40
Yep, worked for SmbFile as well. A transfer that took 372 seconds using the old method, now took 286 seconds :) – Vish Sep 30 '15 at 07:01

score 4 · Answer 2 · answered Dec 06 '12 at 16:39

4

Well since you're using a BufferedInputStream, you aren't reading byte by byte, but rather the size of the buffer. You could just try increasing the buffer size.

answered Dec 06 '12 at 16:39

Zutty

5,357
26
31

score 1 · Answer 3 · answered Dec 06 '12 at 16:40

Reading/writing byte-by-byte is definitely going to be slow, even though the actual reading/writing is done by chunks of the buffer size. One way to speed it up is to read/write by blocks. Have a look at read(byte[] b, int off, int len) method of BufferedInputStream. However it probably won't give you enough of the improvement.

What would be much better is to use nio package (new IO) to copy data using nio channels. Have a look at nio documentation for more info.

score 1 · Answer 4 · answered Dec 06 '12 at 16:50

1

I would suggest to use FileUtils from org.apache.commons.io. It has enough utility methods to perform file operations.

org.apache.commons.io.FileUtils API Here

answered Dec 06 '12 at 16:50

vels4j

11,208
5
38
63

Faster way of copying data in Java?

4 Answers4

Linked