3

I need to copy snapshots of virtual machines running on Proxmox (KVM) servers and copy the snapshots to offsite storage.

Most snapshots are a few gb, but some are rather large, up to 200gb. I would prefer to compress and copy the snapshots in one go, something like:

xz -c dumpfile | scp offsite

The problem is due to the size of the dumpfile and other jobs that needs to run at the sime time this may take 12 hours or more, during which time a lot can happen to interrupt the data stream.

Using rsync I could resume the transfer if it was interrupted, but would then have to compress the whole file before sending it, requiring another 150g on top of the 250g allocated to the snapshot. As storage on the server is at a premium (ssd only) I'd rather prefer not having to allocate that extra disk space.

Perhaps splitting the compressed output out in smaller pieces pkzip style, and transferring those in a queue as they are ready, could be a solution? Tar seems to have an multivolume (-m) option that perhaps could be used. Problem is that the compression process would need to be stopped until the last compressed part had been transferred.

I'm looking for ideas here, not really a concrete solution. I feel i've missed some obvious option. Would prefer using standard Linux software where possible.

Dokbua
  • 1,072
  • 1
  • 10
  • 19

2 Answers2

3

I was going to suggest ChunkFS a fuse filesystem that allows any user to mount a file and see it split as a directory of files, each being a chunk of some given size. You can then rsync these files to the remote and concatenate them there.

However, I note that the same person, Florian Zumbiehl, has also written ChunkSync, which sort of combines the functionality of ChunkFS with that of rsync (without using fuse). It was designed to allow incremental backups of a large file (such as an encrypted filesystem), where only small parts of the large file changes. It effectively splits the file into chunks and backs up each chunk, using hard links for chunks that have not changed since the last backup. You can specify a destination using ssh. Even if making a single backup, it should be useful.

meuh
  • 1,563
  • 10
  • 11
1

A classic case of RTFM, sorry. Carefully reading TFM it turns out that vzdump offers a --compress option that will compress the dump on the fly, so I reduce the required space even more. I then proceed to rsync the compressesd dump off to the backup server and remove the dump from the hypervisor.

The final script became 

vzdump $VZID --mode snapshot --compress lzo --mailnotification failure --mailto my@email.address \
    && rsync -e ssh vzdump-qemu-${VZID}-* backup:/backup \
    && rm -f vzdump-qemu-${VZID}-* \
            || mail my@email.address -s "transfer of $VZID failed" < vzdump-qemu-${VZID}-*.log

~

Dokbua
  • 1,072
  • 1
  • 10
  • 19