Offline Deduplication Software for ZFS

Question

I do have a 300TB Freenas server to backup several linux nodes. Backup works with daily snapshot and rsync tasks.

The user often move big datasets (2-5TB) between the server - so often big files are getting backed up several times on several server.

Online Deduplication would be too expensive (1,5TB RAM....), so is there any offline deduplication software out there?

I mean, the files do have the same name and often the same access times - fdupes would recognize them as identical with minimal effort...

Does fdupes creates hard links or does it connect the data on block level? I think I am searching for a tool for the second option (like online deduplication...). — philipp, Aug 20 '19 at 17:32
No they are not, you cant hard link across file systems where as zfs dedup works at the pool level, so can affect any dataset on the pool where dedup is enabled, be it a mounted dataset or zvol. — krad, Feb 10 '20 at 09:32

score 0 · Answer 1 · answered Feb 18 '20 at 16:55

0

Try BorgBackup - but you will have to use it instead of rsync.

answered Feb 18 '20 at 16:55

vermaden

score 0 · Answer 2 · answered Jun 05 '21 at 10:00

0

You can easily use, in this case, zpaq or zpaqfranz

The latter (my fork) have specific functions to "freeze" or "consolidate" zfs' snapshot in a single archive

answered Jun 05 '21 at 10:00

Franco Corbelli

2 Answers2