1

I want to set up standard rdiff-backup to remote server. However the data is very big for the capacity of internet connection. Therefore I need some way to do the initial copy of data without rdiff-backup (as there is no way to resume initial transfer which failed after hours of thransfer).

My idea was to transfer data initially on physical media to remote server. After that transfer I synchronized the data by rync to make sure they are identical. Now I want to do intial rdiff-backup. When I run the rdiff-backup there are lots of lines

Getting delta (with hash) of XY

as if the directories were NOT identical. What exactly do these lines mean? Do they mean that the remote rdiff-backup decided that the file is not identical, so there is a delta diff transferred?

Is there any other way to do initial transfer of large data which can be resumed in case of failure (as opposed to be restarted from beginning)?

gorn
  • 65
  • 9
  • Did you preserve timestamps and permissions when you seeded the original system? Rsync will start looking for deltas if the metadata doesn't match up. – SmallClanger Apr 24 '13 at 11:13
  • That is a good idea, thanks, however here is what is strange and what happened several times: I started rdiff-backup and it starts getting these deltas. Than as the process is very long it fails in some point (probably network issues), so I delete rdiff-backup-data and try again. And it starts getting these deltas ALL OVER AGAIN, from the first file. Does not it have synchronized the files in the first failed attempt? Or it first gets all deltas and than applies them? – gorn Apr 24 '13 at 11:19
  • I haven't used rdiff-backup, only rsync istelf, so I can't comment directly on that, but my guess is that it's trying to maintain consistency of the destination. Either all of the backup succeeds or none of it does. Likely the new files are written to temporary folders and only copied in to place locally once the sync completes successfully. – SmallClanger Apr 24 '13 at 13:21
  • @SmallClanger: If you are right than there is not hope one can "resume" the initial backup in any way :( – gorn Apr 24 '13 at 15:55

2 Answers2

1

I use duplicity instead of rdiff-backup but from my understanding your problem is here:

My idea was to transfer data initially on physical media to remote server. After that transfer I synchronized the data by rync to make sure they are identical. Now I want to do intial rdiff-backup.

Solution that works

The simplest solution is to rdiff-backup to that physical media at the backup source, instead of copying and rsyncing data. Then transfering the backup via physical media to remote server and use it as a backup directory for remote backups. Because when copying and rsyncing there is still no initial backup and the program needs to crosscheck all files and save the metadata. Thus the Getting delta.... Which is probably not transfering whole files, just a lot of checksums.

Solution that does not work

It may be worth trying to rdiff-backup from the physical media into a directory and then remotely rdiff-backup there from remote server. Though if rdiff-backup uses some local metadata as well (duplicity does), you (or it) will have to transfer that back or recreate it.

Editted according to what worked for OP.

Fox
  • 3,977
  • 18
  • 23
  • Smells right. My money is on "Wrong way to pre-seed" as well. From the [man-page](http://www.nongnu.org/rdiff-backup/rdiff-backup.1.html): Note that you should not write to the mirror directory except with rdiff-backup. – StackzOfZtuff Apr 22 '15 at 08:43
  • I am in situation where I have the files in remote (without rdfiff info), so I tried, to rdiff-backup it "locally" on remote from one dir to another (to create rdiff files) and than rdiff backup from local to remote. However it is still restarting and sends all deltas. Somehow the rdiff "restart strategy" is spawned anytime there is slightest suspicion that something is wrong. – gorn Apr 23 '15 at 14:17
  • @gorn Did you `rsync -a` to make sure both destinations are identical, including mtimes? Not sure if `rdiff-backup` checks inode numbers as well. If it does, your best bet is to get the physical media back, `rdiff-backup` to it, and then copy the backup directory in the other location. – Fox Apr 23 '15 at 14:47
  • @Fox Your initial suggestion did not work (reason unknown), but your suggestion in comment (sort of) worked. I got some random deltas on some files but only on few ones, so it works now. Do you want to edit your answer accordingly for future reference or should i do it? – gorn Apr 24 '15 at 14:49
  • @gorn my comment suggestion is actually in the second paragraph ("I think you should've done ..."). I will clear it up a little. – Fox Apr 24 '15 at 15:09
  • @gorn does it work for you like this? If not, feel free to edit/suggest... – Fox Apr 24 '15 at 15:51
0

Are you really bound by rdiff-backup or you are OK using plain rsync to do incremental backups?

Here's a small script that keeps 4 incremental backups. You can change it easily to keep 9 or 99 incremental backups or any other number.

It uses the --link-dest option from rsync, creating hard links if files are unchanged.

FROM=$HOME
TO=/var/backups
LINKTO=--link-dest=$TO/`/usr/bin/basename $FROM`.1
OPTS="-a --delete -delete-excluded"
NUMBER_OF_BACKUPS=4


/usr/bin/find $TO -maxdepth 1 -type d
  -name "`/usr/bin/basename $FROM`.[0-9]"|\
  /usr/bin/sort -rn| while read dir
do
    this=`/usr/bin/expr match "$dir" '.*\([0-9]\)'`; 
    let next=($this+1)%$NUMBER_OF_BACKUPS;
    basedirname=${dir%.[0-9]}
    if [ $next -eq 0 ] ; then
             /bin/rm -rf $dir
    else
             /bin/mv $dir $basedirname.$next
    fi
done
/usr/bin/rsync $OPTS $LINKTO $FROM/ $TO/`/usr/bin/basename $FROM.0`

With that script, you can first rsync your data manually, they configure the script to be launched regularly.

mikhailian
  • 201
  • 1
  • 7
  • Thanks for this solution, it obviously has advantage of easy initial setup. I am personally bound to rdiff, but others might find this usefull. – gorn Apr 28 '15 at 01:30