8

I have a local directory of media files on a Linux system, which I synchronise with an Amazon S3 account using an s3sync script. Currently, I'm manually running the s3sync script when I know the media files have been modified.

How would I automatically run the script when files are modified?

I was thinking of creating a cron job to run the script every few minutes, but that seems like an excessive amount of processing, because even if there are no changes, the script still has to scan the entire directory structure, which is quite large.

I also considered incron/inotify, which allows running commands when a specific file or directory changes, but these tools don't seem to automatically support monitoring changes to the entirety of a nested directory. Correct me if I'm wrong, but it seems that incron/inotify can only monitor files they've been explicitly told to monitor. e.g. If I wanted to monitor changes to all files at any level inside a directory, I'd have to write separate scripts to monitor directory and file additions/deletions to update the list of files and directories monitored by incron.

Are there more efficient solutions?

Cerin
  • 60,957
  • 96
  • 316
  • 522
  • As you correctly said the inotify mechanism (APIs) is not recursive for directories. So you would need to create additional watches for subdirectories under a watched directory. Also do remember since one inotify watch will be established per subdirectory, it is possible that the maximum amount of inotify watches per user will be reached. The default maximum is 8192 and can be increased by writing to /proc/sys/fs/inotify/max_user_watches. Perhaps you could look at inotifywatch source code (which handles recursive directories) and write something on your own based upon that. – ghostkadost Dec 08 '11 at 05:10
  • 1
    You can also use **inotifywait** command in your own shell script. The examples provided in man page look very promising. – ghostkadost Dec 08 '11 at 05:29

3 Answers3

3

For this kind of tasks, I'm using fssm gem.

create file watcher.rb

require 'fssm'

FSSM.monitor('/dir_to_watch/', '**/*') do
  update {|base, relative| `your_script` }
  delete {|base, relative| `your_script` }
  create {|base, relative| `your_script` }
end

then:

ruby watcher.rb

Of course you can demonize it, if you want.

sparrovv
  • 6,894
  • 2
  • 29
  • 34
2

Here is a sample scenario you might use instead and utilize simple rsync script.

http://andrewwilkinson.wordpress.com/2011/01/14/rsync-backups-to-amazon-s3/

Basically means using fuse and s3fs ( http://code.google.com/p/s3fs/ ) to mount s3 share as a directory on your local filesystem and use rsync to sync the 2. Simple cron job would do the trick.

Bemehow
  • 21
  • 1
1

Now there is an efficient solution. This was just announced (long overdue):
http://aws.amazon.com/blogs/aws/s3-event-notification/

It is very simple to implement - time to throw out all the ugly cron jobs and list-loops.

Tal Weiss
  • 8,889
  • 8
  • 54
  • 62