Is file modification time reliable for cache invalidation?

Question

I want to invalidate some locally generated cache every time file is modified. Invalidation would happen on running manually some command-line tool (no live watch required)

My approach would be to:

1. First generate cache and store file modification date.
2. On consequent runs check if file modification date is different from the one stored - if yes: invalidate cache, and generate it again.

Is using file modification date enough for this approach to be reliable, or should I also check file shortcuts (some hash function) if file modification date hasn't changed (false positives are not a problem, but cache needs to be invalidated every time file has changed).

File will be shared using VCS and cloud storage like Dropbox or OneDrive.

Question is OS agnostic (i.e. it has to work on every commonly used operating system (Windows, Mac OS, other POSIX compliant)).

Answer 1

Is file modification time reliable for cache invalidation?

I'd say yes and I think you should consider using Make.

Let's say we need to cache the computation of an expensive process: counting all the numbers between 1 and n.

The number n is read from a file input.txt.

The file count.cache contains the sequence of all the numbers between 1 and n.

Given the following structure:

data/
├── input.txt
├── Makefile

And the following Makefile:

count.cache: input.txt
    cat $^ | xargs seq >$@

To start with input.txt is empty. Let's put some number in it:

echo '5'>input.txt

Then let's run make:

make

The output is as follow:

cat input.txt | xargs seq >count.cache

Here's the content of count.cache:

cat count.cache
1
2
3
4
5

Now let's run make again:

make
make: `count.cache' is up to date.

Why? In order to generate count.cache (the target), you need input.txt (the prerequisite). If the prerequisite hasn't changed then the target is still valid.

Let's update input.txt:

echo '10'>input.txt

And let's run make again:

make
cat input.txt | xargs seq >count.cache

By default make outputs the commands needed to generate the target. As you can see make has figured out that it needed to regenerate count.cache as its prerequisite input.txt has changed.

Let's verify that:

cat count.cache 
1
2
3
4
5
6
7
8
9
10

---

make looks daunting at first sight but it is an extremely powerful tool. It's definitely worth investing time to learn it; it will pay off hundred times more!

https://www.gnu.org/software/make/

Answer 2

Redis seems the better choice for you! You can spin up an instance easily via docker and use it as a cache.

It is a new dependency but manage a lot of technicality for you