Copy large files in fastest way possible

Question

I tried to find a way to copy large files in fastest way possible...

import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.ArrayList;

public class FastFileCopy {


public static void main(String[] args) {
    try {
        String from = "...";
        String to = "...";
        FileInputStream fis = new FileInputStream(from);
        FileOutputStream fos = new FileOutputStream(to);
        ArrayList<Transfer> transfers = new ArrayList<>();
        long position = 0, estimate;
        int count = 1024 * 64;
        boolean lastChunk = false;
        while (true) {
            if (position + count < fis.getChannel().size()) {
                transfers.add(new Transfer(fis, fos, position, position + count));
                position += count + 1;
                estimate = position + count;
                if (estimate >= fis.getChannel().size()) {
                    lastChunk = true;
                }
            } else {
                lastChunk = true;
            }
            if (lastChunk) {
                transfers.add(new Transfer(fis, fos, position, fis.getChannel().size()));
                break;
            }
        }
        for (Transfer transfer : transfers) {
            transfer.start();
        }
    } catch (IOException ex) {
        ex.printStackTrace();
    }
}

}

then create this class :

import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.nio.channels.FileChannel;

public class Transfer extends Thread {

private FileChannel inChannel = null;
private FileChannel outChannel = null;
private long position, count;

public Transfer(FileInputStream fis, FileOutputStream fos, long position, long count) {
    this.position = position;
    this.count = count;
    inChannel = fis.getChannel();
    outChannel = fos.getChannel();
}

@Override
public void run() {
    try {
        inChannel.transferTo(position, count, outChannel);
    } catch (IOException e) {
        e.printStackTrace();
    }
}

}

I tested it and the result was very very impressive... but there is a big problem, the copied file is veryyyyy larger than the current file !!!

so, please check it and help me to find the problem, thank you :))

Is `Files.copy(source, destination)` not fast enough for you? Also if the file is on a single hard drive, using more than one thread will ***decrease*** performance. — assylias, Mar 19 '14 at 10:53
Bugs in your code aside, what exactly made you think multithreading would make it faster? Do you have a multithreaded disk? Striped perhaps? — user207421, Mar 19 '14 at 10:58
Threads can't possibly speed up an operation which uses maybe 1% of CPU. — Marko Topolnik, Mar 19 '14 at 10:59
@MarkoTopolnik it can however make system CPU time explode :p — fge, Mar 19 '14 at 10:59
@AvB I just copied a 3GB file with `Files.copy` in slightly under 20 seconds on my average desktop pc. — assylias, Mar 19 '14 at 11:08

score 7 · Answer 1 · answered Mar 19 '14 at 11:15

This is a XY problem. Just use Files.copy().

Look at that and see if this is not fast enough for you:

$ ls -lh ~/ubuntu-13.04-desktop-amd64.iso 
-rw-rw-r-- 1 fge fge 785M Jul 12  2013 /home/fge/ubuntu-13.04-desktop-amd64.iso
$ cat Foo.java 
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.nio.file.StandardCopyOption;

public class Foo
{
    public static void main(final String... args)
        throws IOException
    {
        Files.copy(Paths.get("/home/fge/ubuntu-13.04-desktop-amd64.iso"),
            Paths.get("/tmp/t.iso"), StandardCopyOption.REPLACE_EXISTING);
    }
}
$ time java Foo

real    0m1.860s
user    0m0.077s
sys 0m0.648s
$ time java Foo

real    0m1.851s
user    0m0.101s
sys 0m0.598s

And it could be even faster. God knows why, Oracle doesn't use sendfile(2) even though this is Java 8 and Linux 2.2 has been here for quite some time.

On the machine of any self-respecting developer this program will cache the file in memory, and the OS will write back the dirty FS cache *after* the program ends. Add a time sync after the progream's execution and add the times. — Tassos Bassoukos, Mar 19 '14 at 11:20

slim · Accepted Answer · 2014-03-19T13:01:21.397

Since each loop you increment position by count+1, and you make a Transfer with `(fis,fos,position,position+count), your code will create Transfer objects as follows:

new Transfer(fis, fos, 0,count)
new Transfer(fis, fos, count+1, 2count+1)
new Transfer(fis, fos, 2count+2, 3count+2)
new Transfer(fis, fos, 3count+3, 4count+3)
...

So although you'll create filesize / count Transfer classes, you're asking to transfer (count + 1) * (1 + 2 + 3 + ...) bytes in total.

Furthermore, I don't think FileChannel.TransferTo() works the way you think it does. position specifies the position in the source file where you start reading. It doesn't specify the position to which you write in the destination channel. So even when you get the sizes right, you'll end up with the right size output file, but the contents will be jumbled up in whatever order the threads happen to write them.

You could call outChannel.position() to skip to the right place. It's not clear to me what kind of chaos might happen as multiple threads extend the filesize in this way.

It's fine to experiment, and I encourage you to try this out and benchmark. However the comments are correct that the approach is misguided. There is only one disk, backed by just one filesystem buffer, and having multiple threads fight over it won't make it work faster -- and may make it slower.

You're unlikely to improve on:

 long count = 0;
 long size = src.size();
 while(count < size) {
    count += src.transferTo(count, size - count, dest);
 }

Do also note that it's very difficult to make judgements about the performance of file operations, since the filesystem will be caching both reads and writes, so an awful lot of what you do will just be super-cheap operations on RAM.

Also note that, at least when benchmarking, you're going to need to join() with all the threads you've started, before considering the copy complete.

Copy large files in fastest way possible

2 Answers2