1

Here is an example shell script I'm looking at implementing using inotify:

inotifywait -mrq -e close_write -e create -e delete -e move /var/www | while read file;
do   
    rsync -av /var/www1/ /var/www2/  
done

Let's say rsync takes 30 seconds to complete.

If there are 10 inotify events within 5 seconds, will there be 10 processes of rsync running the same job at the same time?

OR

Will there only be one rsync job that is run 10 times, one after the other?

Dennis Williamson
  • 62,149
  • 16
  • 116
  • 151
Travis Mijat
  • 51
  • 1
  • 6

1 Answers1

4

They will be run one after another unless you background the rsync with a & at the end.

This will run them concurrently which is not what you want:

inotifywait -mrq -e close_write -e create -e delete -e move /var/www | while read file
do   
    rsync -av /var/www1/ /var/www2/  &
done

You probably also don't want it run ten times consecutively. This may be what you're after:

while inotifywait -rq -e close_write -e create -e delete -e move /var/www
do   
    rsync -av /var/www1/ /var/www2/  
done

Also, be aware that running inotifywait recursively on a large /var/www may take a while to set up on each invocation. You may want to limit its scope to only watching active subdirectories or just use cron to periodically run rsync.

Edit:

Try this demo:

$ mkdir inotifytest
$ cd inotifytest
$ echo something > testfile
$ while inotifywait -rq -e access .; do echo "processing"; sleep 3; done    # test 1

Now in another terminal, do this:

$ cd inotifytest
$ for i in {1..10}; do cat testfile > /dev/null; done

In the first terminal you should see "processing" one time. If you do the for/cat loop (which represents files being added or deleted in your directories) again, you'll see it again.

Now in the first terminal, interrupt the loop with Ctrl-C and start this loop:

$ inotifywait -mrq -e access .| while read file; do echo "processing"; sleep 3; done    # test 2

In the second terminal, run the for/cat loop again. This time in the first terminal you'll see "processing" ten times with a three second pause between each one (representing the time it takes rsync to complete. Now if you were to run the for loop several times in rapid succession you might be waiting all day.

Dennis Williamson
  • 62,149
  • 16
  • 116
  • 151
  • Thank you Dennis. Understood, in that case inotifywait must collect and hold each inotify event, waiting for each rsync job to finish. PERFECT!!! – Travis Mijat Jul 19 '10 at 22:01
  • @Travis: see my edit. – Dennis Williamson Jul 19 '10 at 22:06
  • I agree, I plan to only set it up on a few small directories, this was just an example. I understand your first piece of code. You are correct, I really don't want to run it 10 times consecutively, this is a starting point. Unfortunately I don't understand your second piece of code. I have no idea what putting that while in front does. Are you able to elaborate please? – Travis Mijat Jul 19 '10 at 22:16
  • I also noticed you dropped the m flag while adding the while. I definitely DO want to execute indefinitely so I assume they are related. – Travis Mijat Jul 19 '10 at 22:20
  • @Travis: The second example runs `inotifywait` once for each iteration of the loop. It then waits for output. If the `while` sees any output then it runs the body of the loop. Once that's done, it runs `inotifywait` again. That keeps `rsync` from running multiple times if it's already caught one change out of a group. You could pile up quite a backlog if changes occur faster than `rsync` can finish. However, it might be better to run `rsync` more aggressively as in your version and let it determine what has changed and what hasn't. I'll add a demo to my answer for you to try. – Dennis Williamson Jul 19 '10 at 22:53
  • Got it, thanks yet again. I'll probably need something in between the two solutions. I know all the issues regarding performance etc etc so I'll give it some thought and then implement a solution. – Travis Mijat Jul 20 '10 at 18:21
  • What is best practice when you want to copy/rsync individual files because rsyncing the entire folder would take very long (because of many files)? The non-monitor variant would miss files. Did I answer my own question, or is there a better way? – Redsandro Dec 28 '12 at 09:43
  • @Redsandro: You can parse the output of `inotifywait` and use then filename in the `rsync` command. In my examples, I'm discarding that output. – Dennis Williamson Dec 28 '12 at 13:52