How do "correctly" sync millions of files with rsync and inotify?

Question

Assume I have 1 million files, the file list would be something like 20MB. If I add just one file, inotify will tell rsync to transfer the file list and the new file. My network is not good and I might have to limit the bandwidth (--bwlimit).

Multiple files might be added at once and, given the network transfer speed, there may be multiple rsyncs running at once.

Is it worth it? Is there a better way?

score 13 · Accepted Answer · edited Feb 23 '23 at 19:07

13

Your case sounds exactly what lsyncd was made for.
It watches the directory with inotify and spawns an rsync with only the changed file(s).
It aggregates changes for a few seconds.
So when you make 5 changes in one second it will only spawn one new sync process to the destination server(s).

edited Feb 23 '23 at 19:07

Matt

1,559
8
11

answered May 05 '15 at 07:46

faker

17,496
2
60
70

Yeah, I've tried lsyncd and it suits me. – seanlook May 13 '15 at 01:49
@seanlook how is performance with 1 million files? – gcb Mar 02 '16 at 08:22
2

@gcb I got a setup with a pretty decent server and >1 million folders(not that lsyncd watches folders, not files!) and it took ~2 days to even do the full startup sync, the low single-threaded inotify_add_watch performance seeming to be the limiting factor. – Izzy Jan 15 '17 at 22:33

How do "correctly" sync millions of files with rsync and inotify?

1 Answers1

Linked