0
for f in `ssh $SSHCRED "~/file-list.sh"`
do
  rsync --progress --times --partial --append --rsh=ssh -r -h --remove-sent-files $SSHCRED:$f $OUTDIR
done

This is what I do today.

I'd like to instead do this by sending the output from ssh $SSHCRED "~/file-list.sh" directly to rsync. That should be faster, and I can easily abort just one rsync operation.

But I see at least one problem with this: The remote script generates a list of files with absolute paths. For rsync to work, I think I need to prepend the SSH credentials before every path, like this:

user@host:/path/to/file1
user@host:/path/to/file2
user@host:/path/to/file3

Is this possible?

Markus Hedlund
  • 1,127
  • 2
  • 19
  • 33

1 Answers1

6

Generate a list of files and hand them over to rsync with the --files-from=FILE option: 

rsync --your-options --files-from=filelist.txt user@host:/basedir targetdir

This should be much faster than calling rsync for every single file.

Sven
  • 98,649
  • 14
  • 180
  • 226
  • Thanks! It almost works the way I want. The only problem is that it recreates the whole dir structure. So say I want to rsync every file in `/home/backups` to `/var/backups` on my local machine, it will create `/var/backups/home/backups`. Can I avoid this? – Markus Hedlund Dec 02 '11 at 14:14
  • Yes. Make sure the list has only relative filenames (`subdir1/filename1` instead of `/home/backups/subdir1/filename1` and then use `user@host:/home/backups` as from-target for rsync. – Sven Dec 02 '11 at 15:31
  • Oh okay. That makes it more complicated, if I want the remote script to handle what files should be copied from where. Even if it's just from one folder. Maybe I could extract the folder path from the first line, then remove that string from every line. – Markus Hedlund Dec 03 '11 at 10:11
  • Or after the script has run, I could flatten the directory structure created by rsync, and leave a single folder with all files. – Markus Hedlund Dec 03 '11 at 10:13