5

My application use rsync to copy files from network driver to local as cache to improve performance.

But since my application might be run in parallel, so as the rsync, and there could be a case that 2 instance of rsync being run in parallel against the same source and directory.

I've run into a problem that the destination doesn't sync completely (leaving one sub directory empty). so I doubt it is a problem of a rsync, so

  • Is rsync safe to copy files from same source to the same destination in parallel?
  • Is there any option available to ensure the correctness of such operation.

The options I am using is: 

rsync -a --copy-unsafe-links --delete
Baiyan Huang
  • 153
  • 1
  • 5

1 Answers1

2

It's probably safe to run rsyn in parallel, but not efficient, since most likely the files will be transfered twice since rsync makes up the list of files to transfer at the beginning. There is of course always the slim chance that the same file is transfered simultaneously by both rsyncs, but the contents should be the same afterwards.

To prevent double runs, usually I write a script that checks for, and otherwise touches a lockfile which is removed at the end. As an option, you can use a trap in your script to clean up the lockfile if you break off the script or when an error occurs (use 'help trap' within your shell). As far as I know, there is no such locking mechanism in rsync itself (which would be hard to implement anwyay).

JvO
  • 561
  • 2
  • 10
  • so you mean rsync is safe to run in parallel with same source and destination, and the problem I met may probably caused by something else? (for performance I think we could use --update, and I actually don't care that too much as this should be a rare case and the only thing I care is correctness) – Baiyan Huang Sep 10 '13 at 22:28
  • It's hard to tell exactly why the empty subdirectory appeared; perhaps it was already created on the remote side by the 'first' rsync with a newer timestamp than the source, which confused the 'second' rsync. Or vice versa. However, if a file gets transfered its contents should be the same. Rsync first creates a temporary file on the destination side (unless you use --inplace), and if everything is transfered it renames the temporary file to the real destination. – JvO Sep 10 '13 at 22:41