Detect changed files in a directory (java 8) from digest or checksum

Question

I'm looking for a simple way to detect if files have changed in a directory between reboots to avoid unnecessary synchronization. What would be the simplest way to do this in java 8 libraries? Should I XOR the md5 digest of each file or XOR the checksums of each file?

ATM we don't need to handle going into sub directories.

Also we should not be using an OS event to detect this change as the method to detect will only be called at startup. The number of files in the directory can change between different versions of the application but these files will not generally change between reboots.

This looks like a relevant post: https://crypto.stackexchange.com/questions/1368/is-it-a-good-idea-to-use-bitwise-xor-on-a-set-of-md5-sums

Answer 1

It depends what you mean by "simple".

On the one hand, you could make use of the file timestamps. But the problem is that timestamps can be misleading:

• Checks depending on time stamps could be affected by clock skew issues. (It depends which clocks are involved, and on how clocks are managed.)

• It is possible for file timestamps to be reset (e.g. by the "root" user) making it appear that a file has not changed.

• It is trivial to change a "modified" file timestamp without actually changing the file; e.g. touch.

On the other hand, if you use checksums you have other problems:

• Computing a file checksum entails reading the entire file. (A partial checksum is not sufficient to detect changes, in general.) Some checksum algorithms are relatively expensive as well.

• You also need to know what the previous checksum for the file was. That means that you need a way / place to store it. That could be just another file, but then you need some infrastructure to update that file (reliably) as part of the synchronization procedure.

• XORing multiple checksums has the problem that you then don't know which files have changed. If one file changes you need to synchronize all of them.

• It is theoretically possible for a file to change and the MD5 checksum to be the same: probability 1 in 2^128. You can probably discount this ... unless yours is a security critical application. (Note that MD5 collision attacks are practical in some contexts; see https://en.wikipedia.org/wiki/Collision_attack)

---

The other thing is that I suspect that you are trying to solve a solved problem. For example, the Linux / Unix rsync utility has options to use either timestamps or (MD5) checksums to decide which files need to be synchronized.

You don't need to implement everything yourself (in Java).

In response to your "we don't have access to the old file tree" there is an easy solution to that. Each time you reboot:

1. copy the file tree
2. compare the current files against the copy that you made last time you rebooted.

Like I said in a comment use your imagination.

Answer 2

Is the file's modified time useful in your situation? Md5sum is a preciser way for some situations.

Answer 3

Here's a routine I'm looking to generate a hash from all files in a directory.

/**
 * Recursively compute a single md5 signature of all files in a directory. Is typically 
 * used to determine if a file in a directory or any of the sub directories have been 
 * modified since the last digest was taken.
 */
public class DirectoryDigest {

    MessageDigest md = null;

    public DirectoryDigest() {
        try {
            md = MessageDigest.getInstance("MD5");
        } catch (NoSuchAlgorithmException e) {
            ApplicationManager.logStackTrace(e);
        }
    }

    public void update(Path dirPath) {
        update(dirPath, null);
    }

    public synchronized void update(Path dirPath, String extension) {

        try {
            Files.newDirectoryStream(dirPath).forEach(file -> {
                if (!Files.isDirectory(file) && file.getFileName().toString().endsWith(extension)) {
                    if (extension != null && !file.getFileName().toString().endsWith(extension)) {
                        System.out.println("not processing: " + file.getFileName());
                        return;
                    }
                    try {
                        byte[] bytes = Files.readAllBytes(file);
                        md.update(bytes);
                    } catch (IOException e) {
                        ApplicationManager.logStackTrace(e);
                    }
                } else {
                    update(file, extension);
                }
            });
        } catch (IOException e) {
            ApplicationManager.logStackTrace(e);
        }
    }

    /**
     * Returns md5 digest signature and resets the digest object.
     * @return
     */
    public String digest() {
        return String.format("%032X", new BigInteger(1, md.digest()));
    }
}

It is used like this:

DirectoryDigest dd = new DirectoryDigest();
dd.update(csConfigDirPath, ".xml");
String currentPeripheralHash = dd.digest();