0

I'm trying to set up rsync to send incremental backups to a remote server. The first backup would go to a "backup" folder, then the next backup would send only the changes to a "backup.1" folder, and so on.

I managed to do this locally with the following command, which seemed to be working as described, creating a backup.1 folder on the second sync : 

rsync -zaP folder_to_backup /backup

I then set up a ssh key pair and managed to get rsync working remotely, so I'm now using : 

rsync -zaP folder_to_backup myuser@myserver:/home/myuser/backup

The sync does work and the files appear on the remove server. But once I run it a second time, the new files simply get added to the existing "backup" folder, rather than creating a backup.1 folder. I also tried other commands with the -b argument, such as : 

rsync -zaPb folder_to_backup myuser@myserver:/home/myuser/backup
rsync -aPb --backup-dir=`date +%s` folder_to_backup myuser@myserver:/home/myuser/backup

But it acts the same in all case. In the last case, the sync still goes to the "backup" folder, the backup-dir argument seems to be ignored completely.

What am I doing wrong?

Edit : Reading the comments, it's possible I got confused somehow when I say "which seemed to be working as described, creating a backup.1 folder on the second sync". That's how I remember it but apparently it's not a feature of rsync?
Instead, I now installed rsnapshot, which is great for incremental backups.

Dino
  • 13
  • 5
  • Hi @Dino, I tried to reproduce your usage of rsync in the first command and I don't get the result you describe. Are you using this inside a script? if that is the case would you post it? – Jorge Valentini Nov 20 '18 at 01:53
  • @JorgeValentini No, I'm using regular rsync. I checked my shell history and the exact command was "rsync -zaP www/ backup.bak". When I ran it the second time, about one hour had passed and a lot of files had changed in www. They were sent to a backup.bak.1 folder. – Dino Nov 20 '18 at 02:27
  • Also, what version of rsync? `rsync --version |grep version` Also, are we sure that does what you think it does? `--backup-dir=DIR make backups into hierarchy based in DIR` – Tim Nov 20 '18 at 02:28
  • Nope @Dino... I cannot replicate the behavior you describe. Just a hunch, but may that be a version thing? did you try the `rsync` locally on `myserver`? – Jorge Valentini Nov 20 '18 at 02:43
  • Investigate `--link-dest` – wurtel Nov 20 '18 at 10:42
  • Why not just use rdiff-backup? That's what it does. And it does it better than anything you'll end up implementing on your own. https://rdiff-backup.net/ – Grant May 24 '22 at 19:41

1 Answers1

0

I wrote this script which uses --link-dest to transfer only new files, while hardlinking all unchanged files. By that only the transfer is incremental while each backup itself is everytime a full back. Example:

src_path="/home/user“
dst_path="user@server:/home/Backups"
rsync_options=(
  --archive # same as --recursive --links --perms --times --group --owner --devices --specials
  --human-readable # output numbers in a human-readable format
  --itemize-changes # output a change-summary for all updates
  --exclude="[Tt][Ee][Mm][Pp]/" # exclude dirs with the name "temp" or "Temp" or "TEMP"
  --exclude="[Tt][Mm][Pp]/" # exclude dirs with the name "tmp" or "Tmp" or "TMP"
  --exclude="Cache/" # exclude dirs with the name "Cache"
)
empty_dir="/tmp/empty_dir"
new_backup="$(date +%Y%m%d_%H%M%S)"
last_backup=$(rsync --dry-run --recursive --itemize-changes --exclude="*/*/" --include="[0-9]*/" --exclude="*" "$dst_path/" "$empty_dir" | grep -oP "[0-9_/]*" | sort -r | head -n1)
last_backup="${last_backup/$(echo "$dst_path" | grep -oP "^.*:")/}" # remove ssh login
rsync "${rsync_options[@]}" --stats --delete --link-dest="$last_backup" "$src_path/" "$dst_path/$new_backup"

Of course this works only if the target filesystem supports hardlinks, but most of them should.

I hope rsync will support --reflink in the future to reduce storage usage for small changes in huge files.

mgutt
  • 503
  • 1
  • 7
  • 24