3

I've heard many times that ZFS snapshots are "free", that because of ZFS' copy-on-write design snapshots do not consume any disk space unless data is changed, and only then the space usage is limited to the blocks which actually changed. Many answers on Stack Exchange, and posts on the internet at large, make this claim or say that space used is "negligible"

How accurate are these statements? How much disk space do ZFS snapshots actually use?

Josh
  • 9,190
  • 28
  • 80
  • 128

1 Answers1

5

I did a test, actually for an un-related problem (zfs send being unable to send a filesystem with over 42,000 snapshots) and to my surprise, I discovered that zfs snapshots actually consume a handful of megabytes. In my tests, this was about 4 MiB/snapshot.

To test this, I created an empty zpool with a single filesystem with no files, directories, or other data at all. I then attempted to create 100,000 snapshots and to my surprise, this faied after creating only 50,698 snapshots:

root@test:~# zpool create tank nvme-nvme.15ad-564d57617265204e564d455f30303030-564d77617265205669727475616c204e564d65204469736b-00000002
root@test:~# zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
tank   199G   116K   199G        -         -     0%     0%  1.00x    ONLINE  -
root@test:~# zfs create tank/test
root@test:~# zfs list
NAME        USED  AVAIL     REFER  MOUNTPOINT
tank        118K   193G       24K  /tank
tank/test    24K   193G       24K  /tank/test
root@test:~# ls -lR /tank/
/tank/:
total 1
drwxr-xr-x 2 root root 2 Mar 14 15:16 test

/tank/test:
total 0
root@test:~# find /tank
/tank
/tank/test
root@test:~# for i in {0..100000}; do zfs snapshot tank/test@snap$i; done
cannot create snapshots : out of space
cannot create snapshots : out of space
...
cannot create snapshots : out of space
cannot create snapshots : out of space
root@proxmoxtest:~# zfs list -t snapshot | wc -l
50698
root@proxmoxtest:~# zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
tank   200G   193G  6.23G        -         -    84%    96%  1.00x    ONLINE  -

Based on this test, it seems that ZFS needs about 3.9 MiB of space for storing metadata for each snapshot on a 200GiB zpool. This appears to vary by pool size; when I tested with a 20GiB zpool, it came out to ~1.8 MiB/snapshot.

So while zfs snapshots may use a "negligible' amount of space (definition of "negligible" may be opinion-based) that cost is not zero; there is a small space overhead (measurable in megabytes) for each snapshot, even when no blocks change.

Josh
  • 9,190
  • 28
  • 80
  • 128
  • What version of ZFS is this? – Michael Hampton Mar 14 '21 at 19:49
  • @MichaelHampton `zfs-0.8.5-pve1` (ZFS on Linux) on proxmox – Josh Mar 14 '21 at 19:54
  • Thanks. It'd be nice to know more details about how the overhead of a snapshot is affected by the size of the filesystem, the size and configuration of the pool, and other variables. You've done 20 and 200 GiB. What about 2TB or 500GB or even 20 or 200 TB? :-) – Andrew Henle Mar 16 '21 at 22:23
  • Agreed @AndrewHenle and I would gladly accept an answer with more details, ZFS internals, or other goodies! For now, I was sharing information I discovered, as it surprised me. If I learn more, I will update my own answer. – Josh Mar 16 '21 at 22:24