3

I have mounted a remote windows share (that will be where my backups to tape will be archiving).

I have 70GB of data that doesn't change that much, so I want to use RSYNC to mirror the data.

/usr/bin/rsync -rlptDv -e ssh --delete \
      --exclude "*Locks" --exclude "tmp" --bwlimit=0 \
      --modify-window=1 /cvs1/* localhost:/mnt/DUBBU01/Linux/Buzz/cvs1/

Now this works fine, in that no files are being updated. To be honest, the folder permissions done mean a damn, as these can be reset if i ever did have to restore from backup.

HOWEVER every single folder gets copied. Not their contents, just the folders. Is there a way to exclude folders containing data, but not the data itself?

The vast number of options in rsync is proving a pain to test this. And with about a million files, and a couple of hundred thousand directories, the ile build can take some time......

Sven
  • 98,649
  • 14
  • 180
  • 226
bomahony
  • 31
  • 1
  • 3
  • One hint, unrelated to your problem: There should be no need to use the `localhost:/mnt/...` syntax with an SSH transport for the target, just `/mnt/../` without the `-e ssh` option should be enough, as rsync can copy onto the same system without any problem. – Sven Nov 02 '11 at 16:44
  • SvenW : That seriously increases the time as it doesnt bother with the delta-transfer algorithm for a quick check. [Or didnt last time i actually looked at rsync for anything remotely like this] – bomahony Nov 02 '11 at 16:48
  • It won't bother with a compare before copying file when it decides it needs to, because reading both files to compare then then writing the differences found to the destination will generally impose more IO load than a straight copy of the whole file. It should not do anything with files that have the same date+time+size unless you have --ignore-times in the options. This should mean that forcing it to do a checksum scan makes the process *slower* for local->local transfers. – David Spillett Nov 02 '11 at 17:46
  • David, I am using "modify-window" as it seems the windows timestamps are to closest two seconds or some such. This would mean that the date isnt actually the same. Also we are talking over two million files and folders. Im pretty sure, building a file list is quicker than a 70GB copy of that many files & folders. So are you also suggesting i remove the "localhost:"? – bomahony Nov 03 '11 at 10:39

4 Answers4

1

What you want to do should be possible with the --relative (or -R) option and a previous run of find to generate a file list: 

find /cvs1 -type f -not \( -name *Locks -o -name tmp \) -print0 > filelist
rsync -pR --modify-window=1 -0 \
      --files-from=filelist /mnt/DUBBU01/Linux/Buzz/

Here you create a null-terminated list of files (only files, not directories) and feed this to rsync as the source for it's operation, informing it about the null-termination with -0. This is useful to avoid problems with spaces etc. in file names.

from the rsync man page:

   -R, --relative

Use relative paths. This means that the full path names specified on the command line are sent to the server rather than just the last parts of the filenames. This is particularly useful when you want to send several different directories at the same time. For example, if you used this command:

rsync -av /foo/bar/baz.c remote:/tmp/

... this would create a file named baz.c in /tmp/ on the remote machine. If instead you used

rsync -avR /foo/bar/baz.c remote:/tmp/

then a file named /tmp/foo/bar/baz.c would be created on the remote machine -- the full path name is preserved.

Sven
  • 98,649
  • 14
  • 180
  • 226
  • Sven Will this increase the time required for this to run? Also, im not sure this is the best option, as there is new files. Wouldnt what you propose only rsyn the files in the search? – bomahony Nov 03 '11 at 10:37
  • It shouldn't increase the run time that much, maybe even be faster, as `rsync` now doesn't have to traverse the file tree itself but works from the list generated by `find`. It then takes every file in this list and compares it to the target tree exactly the same way as before, so it will detect new files, skip files that are the same etc. And yes, it will only sync the files from the `find` command, but this should be the same as when `rsync` does the job, minus the unwanted empty leaf directories. – Sven Nov 03 '11 at 10:47
  • Sven Are you suggesting doing a find right before each rsync? I am a little confused as to what you are implying to be honest. Also some of the files could easily change, so it would still need to use the rsync algorithm for the checks, before copying. I am going to try davids suggestions in regards to folders – bomahony Nov 04 '11 at 16:42
1

Does it actually matter that it considered doing something to the directories each time? I've noticed this behaviour with some of our rsync backups from CIFS shares, but ignored it as the worst affect it has is that there are extra lines in the log files that we need to scan through should there be a problem to investigate. It doesn't (in our case) result in any notable extra data transfer as none of the files get touched unless they themselves have been modified, and if the remote folders are getting acted upon at all the most that is happening is a setting of ownership/perms/dates which is not going to cause significant IO load so take much time.

Edit: As an alternative to just ignoring them, you could filter them out of the output by piping it through grep -v /$, as directories in the log have trailing path separators and files don't. Not ideal, but it will remove the excess output from sight until you find a better solution.

Also, looking at our most recent logs to verify that grep command I notice that it isn't including all directories, just those that have had content within them change (and a few that don't but not many), in our case. The two differences between our rsync options and the ones you are specifying is we are not preserving permissions (no -p/--perms) and are using a larger --modify-window (10 seconds rather than 1). It might be worth trying the --itemize-changes option to see if that offers a clue as to why it is wanting to touch every directory.

David Spillett
  • 22,754
  • 45
  • 67
  • David: The cronjob goes out to a group of people who are basically being spammed with this big list of folders. They should just be seeing the changed files. I know i could reduce verbosity, but that wouldnt list the new/changed files. Also we are talking 500,000 folders here, so its a lot of spam. And the few bytes of data for each folder is still transferred. this is also happening EVERY run of rsync, not just now and again – bomahony Nov 03 '11 at 09:09
  • If the average extra transferred per directory is as much as 50 bytes, that would only be ~25Mbyte (it may be more than that if you have a deep structure or generally long names, of course). You could filter the lines out of the output (see question edit). Also, try `--itemize-changes` to see if that give a clue as to why it is seemingly touching every directory. – David Spillett Nov 03 '11 at 10:47
  • The data transfer is about 80Mb. While not a lot it is annoying. I will kick one off tomorrow with itemize-changes and have a look at what it gives. In regards to your question edit: filtering them out could be an option, but i would rather get it to just stop acting the ass :D I am going to try without permissions (as these are based on the folder in the root of the rsync'd directory). I will also try the lartger modify window. – bomahony Nov 03 '11 at 15:25
0

I had the same problem (folders were listed in the output when I was running rsync). The itemize changes indicated that the permissions were being updated each time, and I eliminated the problem using the --no-p option (I was using rsync -avz). In your case, the permissions aren't important, so I expect you can simply use -rltDv rather than -rlptDv.

Steve Kroon
  • 241
  • 2
  • 8
0

I think you want the rsync option '--prune-empty-dirs'

OpenCoderX
  • 237
  • 2
  • 7