ssh & tar to incrementally transfer (like rsync)

Question

In a hosting environment that has no rsync client, I'd like to perform an action similar to what rsync does: send only those files that have changed.

I would perform this one-liner on the current directory. This updates the archive.tar.gz with changed files, then scp the file to remote site and finally unpack it leaving synced directories. I have the vision of this using multiple pipes but I suppose I'll need an archive to actually use the incremental feature.

I'm getting lost on the tar syntax when using incremental.

Show what you have so far. – Dennis Williamson Nov 02 '09 at 22:46 — Dennis Williamson, Nov 02 '09 at 22:46

score 2 · Answer 1 · answered Nov 02 '09 at 23:12

bozosync.sh

Hmm, if you only want to move the changed files, why would the tar command be updating the archive? If only changed files are being sent, wouldn't the tar archive consist only of them, and so be necessarily recreated from scratch each time?

If I'm understanding this correctly you want something like this, I call it bozosync.sh:

TarFile=fun.tgz
TheScript=$0
result=
ls -1 . | (
  while read f; do
    if [ -f "$f" -a $f -nt $TheScript ]; then
      result="$result $f"
    fi
  done
  tar cvfz $TarFile $result
  touch $TheScript
)

This puts every plain file in the current directory in the tar archive, for each file that is newer than the script itself, and then it runs touch(1) on the script to update its mod time. This does have a race condition, in that a file that is modified in between the test and the touch will be ignored the next time around. If this is a problem you could get around that by copying the script prior to the loop, and then moving it back to itself at the end. (Then you still have a race but the failure is softer, it may simply include a file that hasn't really changed.)

score 2 · Answer 2 · answered Nov 03 '09 at 03:39

This is a modified version of DigitalRoss's script:

#!/bin/bash
TarFile=fun.tgz
TheScript=$0
saveIFS="$IFS"
IFS=$'\n'
files=($( find -maxdepth 1 -type f -newer $TheScript -print ))
tmpfile=$(mktemp)
IFS="$saveIFS"
tar cvfz $TarFile "${files[@]}" && touch -r $tmpfile $TheScript
rm $tmpfile

This reduces the chance of race conditions by saving the result of find in an array and immediately creating a temp file. But that's no guarantee, of course. Also, this script avoids using ls. Another advantage is that it only updates the time stamp of the script if the tar is successful. It will also handle filenames with spaces.

score 0 · Accepted Answer · answered Nov 15 '09 at 20:18

I made a few changes after testing, but this works great thanks.

#!/bin/bash
TarFile=transfer-me.tar.gz
TheScript=$0
saveIFS="$IFS"
IFS=$'\n'
files=($(find . -maxdepth 1 -newer $TheScript -print))
unset files[0]; #remove "."
tmpfile=$(mktemp temp.XXXXXX)
IFS="$saveIFS"
elements=${#files[@]}
if [ "$elements" -gt "0" ]; then
    tar cvfz $TarFile --exclude ".svn" --exclude ".swp" --exclude "Icon?" --exclude ".DS_Store" --exclude "$TarFile" "${files[@]}" && touch -r $tmpfile $TheScript
fi
rm $tmpfile;

ssh & tar to incrementally transfer (like rsync)

3 Answers3

bozosync.sh