I'm monitoring a large directory of 500GB - 1TB with a file watching service that fills my database. During an initial scan, starting up the scanning and native watching events (add file, add directory), without even any postprocessing (metadata, thumbnails, database checks) can take some 2-3 hours to complete
When the watcher restarts (eg. crash or update) it has to redo the initial scan, during which I have two options:
Full scan - process everything again
- The watcher would again look through all the file data and compare it with database entries.
- If any found different or new, the database would be updated.
- Speed : Very slow (~5-6 hours)
- Lost data: None, most robust
.
Oblivious scan - ignore the initial scan's events
- The watcher would initialise the watching correctly
- but wouldn't be able to register any new files or changes between the beginning of restart and completion of the initial scan
- Speed: Roughly a half time of the full scan (~2-3 hours)
- Lost data: A lot, not really reliable (unless run at night when no changes can happen)
.
Furthermore, I considered a third option, but not really sure whether it's doable:
Backup and restore
- During the run, use a second process to backup the watcher's data and any objects responsible for watching
- On restart, instead of re-running the initial scan, restore the watching using the data stored in the backup process
- Then continue working normally
- Speed: Few seconds needed to restart the watcher process
- Lost data: Probably none, unless data change happens exactly during the few seconds of restart (good enough for my case)
.
I'm wondering if you have ever tried the backup-restore of a watcher service? Or do you have any thoughts about it?
.
(In my case the watcher uses node.js, chokidar, fs.watch, FSWatcher and some native C fs_event_wrap)
