How to tell when a File is "Done" copying into a watched Directory?

Question

I'm using the WatchService API to watch a directory, and getting ENTRY_CREATE events when a user starts copying a file into the directory. The files I'm working with can be large, though, and I'd like to know when the copy is finished. Is there any built in java API I can use to accomplish this, or am I best off to just keep track of the created files' size and start processing when the size stops growing?

EDIT: Here is my example code:

package com.example;

import java.io.File;
import java.nio.file.FileSystems;
import java.nio.file.Path;
import java.nio.file.StandardWatchEventKinds;
import java.nio.file.WatchEvent;
import java.nio.file.WatchKey;
import java.nio.file.WatchService;

public class Monitor {
    public static void main(String[] args) {
        try {
            String path = args[0];

            System.out.println(String.format( "Monitoring %s", path  ));
            WatchService watcher = FileSystems.getDefault().newWatchService();
            Path watchPath = FileSystems.getDefault().getPath(path);

            watchPath.register(watcher, StandardWatchEventKinds.ENTRY_CREATE, StandardWatchEventKinds.ENTRY_MODIFY);

            while (true) {
                WatchKey key = watcher.take();
                for (WatchEvent<?> event: key.pollEvents()) {
                    Object context = event.context();
                    System.out.println( String.format( "Event %s, type %s", context, event.kind() ));

                }
            }

        } catch (Exception e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
    }
}

Which Produces this output:

Monitoring /Users/ericcobb/Develop/watch
Event .DS_Store, type ENTRY_MODIFY
Event 7795dab5-71b1-4b78-952f-7e15a2f39801-84f3e5daeca9435aa886fbebf7f8bd61_4.mp4, type ENTRY_CREATE

Answer 1

When an entry is created, you will get an ENTRY_CREATE event. For every subsequent modification, you will get an ENTRY_MODIFY event. When copying is completed, you will be notified with an ENTRY_MODIFY.

Answer 2

The only valid option to ensure, no other process is writing to the file anymore is to successfully obtain a Write lock on the file. Although a bit clumsy, you could use FileChannel and FileLock for that purpose

try(FileChannel ch = FileChannel.open(p, StandardOpenOption.WRITE);
    FileLock lock = ch.tryLock()){

    if(lock == null) {
        //no lock, other process is still writing            
    } else {
        //you got a lock, other process is done writing
    }
 } catch (IOException e) {
     //something else went wrong
 }

You could as well obtain a lock using ch.lock() which is a blocking call. But since ENTRY_MODIFY events are continously generated when the file is being written to, you can as well use tryLock and wait for the next event.

Answer 3

• I think that the most common way to accomplish this is to copy the file into a temporary directory on the same partition and then move it to the monitored directory by an atomic move command.
• Perhaps a solution could also be achieved by playing around with the files time stamps

However both of these solutions need to be realized in the writing application. I think you cannot tell surely from the reading application.

Answer 4

This is the library that we use today:

https://github.com/levelsbeyond/jpoller

It's a fork of an open-source project that the original author no longer maintains. We've fixed a few bugs in it, and today use it in production.

It works by checking the size of files in a directory, and considers the file done (ready for processing) once the file size stops growing for a period of time.

Answer 5

If you have the ability to control the file writes, you can write to a temp file, then rename it to the target. That should generate a single ENTRY_MODIFY.

Something like:

Path tempFile = Files.createTempFile("tmp", "tmp");
writeSomeContents(tempFile);
Path target = Paths.get(targetFilename);
Files.move(tempFile, target);

I recommend waiting a few MS after receiving the notification, to wait for the move to complete.

Answer 6

Maybe you can tell by trying to acquire write access to the file, but this depends on two factors and requires research:

1. The writing application must acquire exclusive write permission to the file and mustn’t close the file before it is in fact complete.
2. It must be possible to test for this from Java. I would give it a try whether opening the file as FileChannel for WRITE access throws an exception if the file is still locked for writing by another program. Unluckily, the Javadoc doesn’t go into detail about this.

Answer 7

File file = child.toAbsolutePath().toFile();
if(file.isFile() && file.isWrite())
{
}

maybe this would be the way to find out what you want but not in the while statement.

It will work when you really access to file.