Synchronize zfs pools fast

Question

My server saves incremental backups on a zfs volume. Due to the data being very similar I can reduce the growth significantly - there is around 500GB of "new data" every day but the pool grows only around 5-10GB/day, the rest is stored in deduplication / compression.

I want to copy the backup to encrypted usb disks, which I also set up as zfs volumes for that reason. When I sync the backup either with rsync or zfs send/recieve it seems like all data is transfered again (just stored as dedup on the usb drive). Due to this, the backup currently takes over 24h, which makes daily backups impossible.

Is there a way to backup this faster?

@MichaelHampton I already tried that: `zfs send history/backup@daily | pv | zfs recv backup1/history` this one still sends the 500GB, but does not transfer that much. It still is not faster than transferring the 500GB — user19760, Oct 31 '16 at 20:04

score 3 · Accepted Answer · answered Nov 02 '16 at 09:05

Michael Hampton's advice is spot on and the Solaris man pages are quite good, but the concept is not that easy to grasp for a beginner. I will outline the points I've experienced when writing my scripts.

Essentially, you first do a snapshot x and a full send/recv as usual:

# Initial send, destroy all filesystems on the destination
# pool which are not present on the source pool.
zfs snapshot pool0@snap0
zfs send -R pool0@snap0 | zfs recv -Fdu pool1

After that, you do snapshot x+1 and send it incrementally. You can delete old snaps on the source, but you need to keep the last on (most recent one) so that differences can be calculated. If you lose/destroy your last snapshot on the source, you will have to start over with a full initial send!

# incremental send, destroy all filesystems on the destination
# pool which are not present on the source pool. Afterwards, old
# snapshots can be destroyed.
zfs snapshot pool0@snap1
zfs send -R -I pool0@snap0 pool0@snap1 | zfs recv -Fdu pool1
zfs destroy pool0@snap0

# Afterwards, repeat and replace snap1 with snap2 and snap0 with snap1 etc.

Some advice from my own experiences:

Deleting the last snapshot means you need to start over, so be careful, check for successful return values in your script first.
You can either number the snaps or use date - numbering is easier, but dates are better if you look at the logs and/or do frequent snapshots.
When experimenting with options and trying -nv to simulate, be advised that this works for send, but will fail with recv, because there is nothing to receive. This is not obvious from either the manual or the error messages.
Snapshots will take up space until they are destroyed, and they "lock" your deleted space. If you do backup frequently, this is not a problem. If you use more than one target disk/pool and/or backup seldom, you may experience disk space constraints. In illumos/OpenZFS there exists the bookmarks feature, which can be a way to circumvent this problem. Unfortunately, it currently only supports single snapshots instead of recursive, so you have to add the recursive code yourself.
If you do not want to use/write your own, use one of the many ones on Github. I think znapzend is the most mature one of them.

Synchronize zfs pools fast

1 Answers1