36

I'm trying to syncronize files from a remote server that is not reliable, meaning the connection tends to fail "randomly" with

rsync: connection unexpectedly closed

Rsync is called with --partial, so I'd like to be able to call rsync in a loop until files are fully transfered. There doesn't seem to be a flag to tell rsync to retry.

What would be the best way to script it? A bash for loop?

Bruno Lopes
  • 615
  • 2
  • 6
  • 8
  • 1
    While the answers below are helpful, it might be a better idea to find out why it's failing and make it stop. I assume you control the server, the client and the network in between, so you can run tcpdump at various places to see how it was closed. Perhaps one of your firewall or VPN devices is closing the connection? – MarkR Jan 02 '10 at 17:30
  • The server it's connecting to is a shared host, so I don't have root access to it. I might look into why it is closing, but for now I'm more than happy to just have it retry. Thanks anyway for the heads up – Bruno Lopes Jan 03 '10 at 02:45
  • 1
    I was shocked to discover rsync didn't actually have the ability to retry! – Michael Apr 13 '15 at 16:36
  • rsync --partial --append does have the ability to retry, manually, or in a script checking the return code. Duplicate of: http://superuser.com/questions/302842/resume-rsync-over-ssh-after-broken-connection – rickfoosusa Jan 23 '16 at 19:12

4 Answers4

38

If you are syncing everything in one sync, call rsync in a loop until rsync gives you a successful return code.

Something like:

RC=1 
while [[ $RC -ne 0 ]]
do
   rsync -a .....   
   RC=$?
done

This will loop, calling rsync, until it gives a return code of 0. You may want to add a sleep in there to keep from DOSing your server.

Abdull
  • 187
  • 1
  • 14
David
  • 3,555
  • 22
  • 17
  • 39
    you can just write while ! rsync -a .... ;do sleep 5;done – Justin Jan 01 '10 at 23:35
  • I'm now using the format @Justin mentioned and it works like a charm :) Thanks! – Bruno Lopes Jan 02 '10 at 01:11
  • 1
    just be careful how you use this, for example, if tried to run it via cron then you could run into concurrency issues. – Keiran Holloway Jan 02 '10 at 03:20
  • 4
    This'll turn into an infinite loop if there's a non-transitent error (e.g. a permissions error reading or writing files). I'd recommend checking rsync's exit status for specific network errors, something like `RC=12; while [[ $RC -eq 12 || $RC -eq 30 ]]` (or whatever exit statuses you're seeing from network drops). – Gordon Davisson Jan 02 '10 at 23:09
  • 1
    As I'm not running this (completely) unattended, infinite loops and concurrency are not much of an issue, thank you. – Bruno Lopes Jan 03 '10 at 02:46
  • 4
    It's a shame rsync doesn't have this feature built in, because any kind of shell based loop will prompt for a password in cases where automatic authentication isn't available or appropriate. – Michael Apr 13 '15 at 21:15
  • It is SOOO frustrating to see answers with messy and unnecessary manipulation of `$?` upvoted, and in this case compounded by testing RC even before the first `rsync`. Use "while" how it's intended to be used: `while ! rsync … ; do echo "Retrying..." ; done` (put `true` instead of `echo…` if you don't want the message) – Martin Kealey Apr 05 '23 at 23:09
15

I ran into this same problem a while back. In the end I wrote something similar to David's answer, but gussied it up a little with max retries, responding to Ctrl-C, and such: http://blog.iangreenleaf.com/2009/03/rsync-and-retrying-until-we-get-it.html.

The obvious solution is to check the return value, and if rsync returns anything but success, run it again. Here was my first try:

while [ $? -ne 0 ]; do rsync -avz --progress --partial /rsync/source/folder backupuser@backup.destination.com:/rsync/destination/folder; done

The problem with this is that if you want to halt the program, Ctrl-C only stops the current rsync process, and the loop helpfully starts another one immediately. Even worse, my connection kept breaking so hard that rsync would quit with the same "unkown" error code on connection problems as it did on a SIGINT, so I couldn't have my loop differentiate and break when needed. Here is my final script:

#!/bin/bash

### ABOUT
### Runs rsync, retrying on errors up to a maximum number of tries.
### Simply edit the rsync line in the script to whatever parameters you need.

# Trap interrupts and exit instead of continuing the loop
trap "echo Exited!; exit;" SIGINT SIGTERM

MAX_RETRIES=50
i=0

# Set the initial return value to failure
false

while [ $? -ne 0 -a $i -lt $MAX_RETRIES ]
do
 i=$(($i+1))
 rsync -avz --progress --partial /rsync/source/folder backupuser@backup.destination.com:/rsync/destination/folder
done

if [ $i -eq $MAX_RETRIES ]
then
  echo "Hit maximum number of retries, giving up."
fi
Ian Greenleaf Young
  • 649
  • 6
  • 9
8

purtting it all together with sshpass

while ! sshpass -p 'xxxx' \
  rsync --partial --append-verify --progress \
  -a -e 'ssh -p 22' /source/ remoteuser@1.1.1.1:/dest/; \
  do sleep 5;done
symcbean
  • 21,009
  • 1
  • 31
  • 52
Toolkit
  • 179
  • 1
  • 3
0

Just another take - a simple one liner to run indefinitely until succeeded, with a 1 second delay after a failure.

while true; do rsync -avz --info=progress2 --partial src/ dsr/ && break || sleep 1; done

And below a slight modification limiting to 10 failures.

for i in {0..10}; do rsync -avz --info=progress2 --partial src/ dsr/ && break || sleep 1; done
too
  • 161
  • 1
  • 4