raidz1 6x4TB = 16.9t?

Question

I an running Ubuntu server with the latest version of zfs-utils. I installed 6x4TB disks (lsblk -b shows all disks partition 1 size=4000787030016) and created a raidz1 configuration with all 6 disks. The raidz calculator website said I should see 20TB "usable". When I run "zpool list" I see 21.8T "FREE". When I run "zfs list" I see 16.9T "AVAIL". When I run "df -h" I see 17T "Size". It is my understanding that "similar" to RAID5. I am quite surprised to see that I started with approximately 24T of disks and after raidz1 I am left with only 17T? It is my understanding that raidz1 is "similar" to RAID5 and expected to lose at least 4T for striping, but where did the other 3T go?

2021/02/20 update - I have deleted and recreated /tank a few times and may have posted stats from a previous build. Below are the stats when using 6x4TB disks. Apologies for the messy display. I am not yet familiar with how to properly enter/display stuff on this forum.

root@bignas1:~# zfs list
NAME   USED  AVAIL     REFER  MOUNTPOINT
tank   805K  16.9T      153K  /tank

root@bignas1:~# zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH ALTROOT
tank  21.8T  1.12M  21.8T        -         -     0%     0%  1.00x    ONLINE  -

root@bignas1:~# zpool list -v
NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
tank       21.8T  1.12M  21.8T        -         -     0%     0%  1.00x    ONLINE  -
  raidz1   21.8T  1.12M  21.8T        -         -     0%  0.00%      -  
ONLINE  
sda        -      -      -        -         -      -      -      -  ONLINE  
sdb        -      -      -        -         -      -      -      -  ONLINE  
sdc        -      -      -        -         -      -      -      -  ONLINE  
sdd        -      -      -        -         -      -      -      -  ONLINE  
sde        -      -      -        -         -      -      -      -  ONLINE  
sdf        -      -      -        -         -      -      -      -  ONLINE  

root@bignas1:~# zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
tank  21.8T   912K  21.8T        -         -     0%     0%  1.00x    ONLINE  -

root@bignas1:~# zfs list
NAME   USED  AVAIL     REFER  MOUNTPOINT
tank   594K  16.9T      153K  /tank
root@bignas1:~# zfs version
zfs-0.8.3-1ubuntu12.6
zfs-kmod-0.8.3-1ubuntu12.5

2021/02/20 update 2

root@bignas1:~# cat /sys/module/zfs/parameters/spa_slop_shift 5

root@bignas1:~# lsblk | grep sd sda 8:0 0 3.7T 0 disk
├─sda1 8:1 0 3.7T 0 part
└─sda9 8:9 0 8M 0 part
sdb 8:16 0 3.7T 0 disk
├─sdb1 8:17 0 3.7T 0 part
└─sdb9 8:25 0 8M 0 part
sdc 8:32 0 3.7T 0 disk
├─sdc1 8:33 0 3.7T 0 part
└─sdc9 8:41 0 8M 0 part
sdd 8:48 0 3.7T 0 disk
├─sdd1 8:49 0 3.7T 0 part
└─sdd9 8:57 0 8M 0 part
sde 8:64 0 3.7T 0 disk
├─sde1 8:65 0 3.7T 0 part
└─sde9 8:73 0 8M 0 part
sdf 8:80 0 3.7T 0 disk
├─sdf1 8:81 0 3.7T 0 part
└─sdf9 8:89 0 8M 0 part

https://wintelguy.com/2017/zfs-storage-overhead.html shows some information on ZFS overhead. — Tero Kilkanen, Feb 20 '21 at 07:17
if /tank is not already used recreate it, seems ro me that you not assigned all available space — djdomi, Feb 20 '21 at 16:12
I did delete and recreate /tank a few times. Here is the latest iteration with all 6x4TB drives. — Gordon Shumway, Feb 20 '21 at 17:01
Can you add the output of `zpool iostat -v` and `zpool status -v`? — shodanshok, Feb 21 '21 at 13:31

score 3 · Answer 1 · answered Feb 20 '21 at 21:26

First, you need to consider than a 4 TB disk really is a 3.64 TiB one (due to decial/binary conversion, ie: one TB is defined as 10^12 rather than 2^40).

So, 3.64 TiB * 5 data disk = 18.2 TiB. Note: by design, zpool list reports the total raw space, without considering parity overhead. This is the reason it shows 21.8 TiB (3.64 * 6)

From that, you need to subtract the space ZFS reserves for itself. By default it is set a 3.2% of total pool size (tunable via spa_slop_shift)

18.2 TiB - 3.2% = 17.61 TB

Curiously, you lose an additional ~600 GB I can not account for. On a test machine of mine configured with 6x 3.64 TiB virtual disks, zfs list reports the expected 17.6 TiB as AVAIL.

Can you show the output of cat /sys/module/zfs/parameters/spa_slop_shift and lsblk ?

score -1 · Answer 2 · answered Feb 20 '21 at 17:37

-1

I am quite surprised to see that I started with approximately 24T of disks and after raidz1 I am left with only 17T?

4000787030016 bytes is roughly 3.64 T, your "thanks" go to marketing.

answered Feb 20 '21 at 17:37

poige

9,448
2
25
52

This does not explain the discrepancy between `zpool list` showing `21.2 T` and `zfs list` showing `16.9 T`, which was the actual question. – Michael Hampton Feb 20 '21 at 21:40
root@bignas1:~# cat /sys/module/zfs/parameters/spa_slop_shift 5 – Gordon Shumway Feb 21 '21 at 01:27
@MichaelHampton how come you being not Gordon tell me what was the actual question. It was your idea about original question — that's fine, but I don't care about it – poige Feb 21 '21 at 04:59
Output of cat command is in the comments. I put the output of lsblk in the original post as it exceeded the max comment size. Thank you. – Gordon Shumway Feb 21 '21 at 17:54

raidz1 6x4TB = 16.9t?

2 Answers2