Efficient way to move directories recursively and merge them in Java

Question

I am looking for the most efficient way to move a directory recursively in Java. At the moment, I am using Apache commons-io as shown in the code below. (If the destDir exists and contains part of the files, I would like those to be overwritten and the nested directory structures to be merged).

FileUtils.copyDirectoryToDirectory(srcDir, destDir);
FileUtils.deleteDirectory(srcDir);

While this does the trick, in my opinion, it isn't efficient enough. There are at least two issues that come to mind:

• You will need to have twice as much space.
• If this is an SSD, copying the data over to another part of the drive and then erasing the old data will eventually have an impact on the on the hardware, as it will in effect shorten the hard disk's life.

What is the best approach to do this?

As per my understanding commons-io doesn't seem to be using the new Java 7/8 features available in Files. On the other hand, I was unable to get Files.move(...) to work, if the destDir exists (by "get it to work" I mean have it merge the directory structures -- it complains that the destDir exists).

Regarding failures to move (please correct me, if I am wrong):

• As far as I understand, an atomic move is one that only succeeds, if all files are moved at once. If I understand this correctly, this means that this is copying first and then deleting. I don't think this is what I'm looking for.
• If a certain path/file cannot be moved, then the operation should cease and throw an exception, preserving the current source path it reached.

Please, note that I am not limiting myself to using the commons-io library. I am open to suggestions. I am using Java 8.

Answer 1

This is just an answer to the "what needs to happen to the filesystem" part of the question, not how to do it with Java.

Even if you did want to call out to an external tool, Unix mv is not like Windows Explorer. Same-name directories don't merge. So you will need to implement that yourself, or find a library function that does. There is no single Unix system call that does the whole recursive operation (let alone atomically), so it's something either your code or a library function has to do.

If you need to atomically cut from one version of a tree to another, you need to build a new tree. The files can be hard-links to the old version. i.e. do the equivalent of

cp -al dir/  new
rsync -a /path/to/some/stuff/  new/
# or maybe something smarter / custom that renames instead of copies files.

# your sanity check here

mv  dir old &&
mv  new dir &&   # see below for how to make this properly atomic
rm -rf old

This leaves a window where dir doesn't exist. To solve this, add a level of indirection, by making dir a symlink. Symlinks can be replaced atomically with mv (but not ln -sf). So in Java, you want something that will end up doing a rename system call, not an unlink / rename.

---

Unless you have a boatload of extremely small files (under 100 bytes), the directory metadata operations of building a hardlink farm are much cheaper than a full copy of a directory tree. The file data will stay put (and never even be read), the directory data will be a fresh copy. The file metadata (inodes) will be written for all files (to update the ctime and link count, when creating the hardlink farm, and again when removing the old tree, leaving files with the original link count.

---

If you're running on a recent Linux kernel, there is a new(2013) system call (called renameat2) available that can exchange two paths atomically. This avoids the symlink level of indirection. Using a Linux-only system call from Java is going to be more trouble than it's worth, though, since symlinks are easy.

Answer 2

I am answering my own question, as I ended up writing my own implementation.

What I didn't like about the implementations of:

• Apache Commons IO
• Guava
• Springframework

for moving files was that all of them first copy the directories and files and then delete them. (As far as I checked, September 2015) They all seem to be stuck with methods from JDK 1.6.

My solution isn't atomic. It handles the moving by walking the directory structure and performing the moves file by file. I am using the new methods from JDK 1.7. It does the job for me and I'm sure other people would like to do the same and wonder how to do it and then waste time. I have therefore created a small Github project, which contains an illustration here:

• FileUtils#moveDirectory((Path srcPath, Path destPath)
• An illustration of how to use it can be seen here.

If anybody has suggestions on how to improve it, or would like to add features, please feel free to open a pull request.

Answer 3

Traverse the source directory tree:

• When meeting a directory, ensure the same directory exist in the target tree (and has the right permissions etc).
• When meeting a file, rename it to the same name in the corresponding directory in the target tree.
• When leaving a directory, ensure it is empty and delete it.

Consider carefully how any error should be handled.

Note that you might also simply call out to "rsync" if it is available on your system.