1

I have a pair of USB drives I use for backing up to via rsnapshot. One drive is left plugged in and the other is taken off site. For the first swap, I ran "rsync -aH" to copy all the contents from current drive onto the offsite drive. This process took 6 full days. I've now reached the point where I'd like to update the offsite drive.

Since rsnapshot "slides" the backups so that daily.0 is always the newest, I did the math (24 new backups since the last rotation) and renamed the existing snapshots to where they'd match.

Then I ran "rsync -aH" again, but it appears that rsync is not detecting that the existing files are linked to the same files as the newer ones and so it's done fresh copies of them. eg, even though the latest in daily.0/server/etc/passwd is the same as the existing daily.30/server/etc/passwd, they have different inodes (but daily.31/server/etc/passwd shares it's inode with daily.30 as expected).

So, on the original rsnapshot created disk currently:

daily.0/server/etc/passwd  inode: A
daily.30/server/etc/passwd inode: A
daily.31/server/etc/passwd inode: A

After the first rsync onto the off-site disk (using "current" names to hopefully make things less confusing):

daily.30/server/etc/passwd inode: A
daily.31/server/etc/passwd inode: A

And after this newest rsync:

daily.0/server/etc/passwd  inode: B
daily.30/server/etc/passwd inode: A
daily.31/server/etc/passwd inode: A

In addition to losing the hard-drive space, it's taking rsync days to verify things are in sync. It's not done yet, but I'm not going to be surprised if this process is also going to take 6 days. :-(

So, what's a better mechanism for refreshing/resyncing these drives? I'd like to stay away from wiping and re-rsyncing because of the amount of time it takes to write the data.

(One additional pain: the drives are not identical and they have other things on them, so just dd'ing them is currently not an option.)

Philippe Chaintreuil
  • 180
  • 1
  • 8
  • Buying a USB3 or controller, and faster drives would probably be the easiest fix. Then just wipe/re-sync. – Zoredache Sep 26 '14 at 16:34
  • You used rsync the first time and rsnapshot the second time. Maybe there was a difference that made the new inodes get created. Try running rsnapshot a second time and make sure new inodes are not created for equal files. (Use a small directory as a test.) – TomOnTime Sep 27 '14 at 04:57
  • @TomOnTime No, sorry if I confused you. The data is rsnapshot created backups, I'm rsync'ing that data onto another drive. The first rsync maintained all the hard links, but then an additional rsync did not maintain the hardlinks between the new files and the old files, even though on the source all those files share the same inode. – Philippe Chaintreuil Sep 27 '14 at 13:38
  • What filesystem are you using? Some don't support inodes and fake it to the operating system. For example, MS-DOS doesn't really have inodes. – TomOnTime Sep 27 '14 at 18:18
  • They're on ReiserFS (3) drives on Linux. – Philippe Chaintreuil Sep 29 '14 at 15:02
  • I run a number of setups with multiple disks swapped daily to cater for backups and never had an inclination to duplicate a backup disk. Similarly in the days when tape was more common, I never duplicated a tape. So I am curious, what is the reason to not just rotate your rsnapshot disks each day, which is what I think most people do. It would also rid you of this issue. – Gordon Oct 19 '14 at 12:49
  • @Gordon One drive is going offsite. It's unscheduled retrieval is very "expensive" for lack of a better word. (Pretend the server is on a ship with satellite internet: I can't physically get to it once it leaves port.) I always want access to the most recent backups, as the only things I've had to retrieve are things that people deleted yesterday or last week. – Philippe Chaintreuil Oct 21 '14 at 23:59
  • Did any UID/GID/perms change? If they changed, rsnapshot will (if configured) create a new file, not a hard link. (I'm also curious if you ever found the solution to this issue) – TomOnTime Nov 30 '14 at 11:32
  • I don't think anything changed, /etc/passwd's metadata settings are pretty static. I have not found a solution to this issue (aside from the just delete everything and do a full sync again). – Philippe Chaintreuil Dec 01 '14 at 12:33

1 Answers1

0

I see you mentioned that it took 6 days to do the original copy over USB. If you are not using USB 3.0, then it would make a massive difference to do so. An add-on USB 3 controller is very cheap too.

Gordon
  • 21
  • 1