I have a 16 core virtualization FreeNAS 11.3-U4.1 instance running on Hyper-V with 32GB of RAM with a 18TB iscsi target on DELL T630 server grade hardware.
HDD are on IT mode from the RAID card, so presented to the FreeNAS box as individual drives.
I do not have any dedupe running, default compression only.
The windows share is running ReFS as a iscsi initiator on Windows Server 2019 and is not full only 1TB or so, used for transferring test movie files.
From multiple machines tested (one at a time), it is initially fast on any transfer - read or write @ 130MB/sec - but then after a minute or so slows down to 40Mbps and at times even keeps stalling.
Network tests are fast both internally via machines on same box and also external machines:
root@freenas[~]# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
^Ciperf3: interrupt - the server has terminated
root@freenas[~]# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.69.139, port 49435
[ 5] local 192.168.69.4 port 5201 connected to 192.168.69.139 port 49437
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 105 MBytes 878 Mbits/sec
[ 5] 1.00-2.00 sec 112 MBytes 941 Mbits/sec
[ 5] 2.00-3.00 sec 112 MBytes 938 Mbits/sec
[ 5] 3.00-4.00 sec 112 MBytes 939 Mbits/sec
[ 5] 4.00-5.00 sec 112 MBytes 941 Mbits/sec
[ 5] 5.00-6.00 sec 111 MBytes 935 Mbits/sec
[ 5] 6.00-7.00 sec 112 MBytes 939 Mbits/sec
[ 5] 7.00-8.00 sec 111 MBytes 933 Mbits/sec
[ 5] 8.00-9.00 sec 112 MBytes 942 Mbits/sec
[ 5] 9.00-10.00 sec 112 MBytes 941 Mbits/sec
[ 5] 10.00-10.00 sec 339 KBytes 955 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 1.09 GBytes 933 Mbits/sec receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.69.5, port 50601
[ 5] local 192.168.69.4 port 5201 connected to 192.168.69.5 port 50602
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 392 MBytes 3.29 Gbits/sec
[ 5] 1.00-2.00 sec 407 MBytes 3.41 Gbits/sec
[ 5] 2.00-3.00 sec 426 MBytes 3.57 Gbits/sec
[ 5] 3.00-4.00 sec 430 MBytes 3.60 Gbits/sec
[ 5] 4.00-5.00 sec 413 MBytes 3.46 Gbits/sec
[ 5] 5.00-6.00 sec 429 MBytes 3.60 Gbits/sec
[ 5] 6.00-7.00 sec 372 MBytes 3.12 Gbits/sec
[ 5] 7.00-8.00 sec 429 MBytes 3.60 Gbits/sec
[ 5] 8.00-9.00 sec 424 MBytes 3.56 Gbits/sec
[ 5] 9.00-10.00 sec 425 MBytes 3.56 Gbits/sec
[ 5] 10.00-10.00 sec 125 KBytes 2.29 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
CPU load via top is minimal:
last pid: 15377; load averages: 1.15, 0.90, 1.06 up 0+12:42:50 03:48:22
44 processes: 1 running, 43 sleeping
CPU: 0.1% user, 0.0% nice, 15.7% system, 0.0% interrupt, 84.2% idle
Mem: 14M Active, 488M Inact, 1060M Laundry, 28G Wired, 1584M Free
ARC: 25G Total, 852M MFU, 19G MRU, 4773M Anon, 580M Header, 75M Other
20G Compressed, 23G Uncompressed, 1.12:1 Ratio
Swap: 6144M Total, 46M Used, 6097M Free
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
104 root 41 20 0 329M 258M kqread 5 3:15 0.77% python3.7
15347 root 1 20 0 7892K 3452K CPU5 5 0:00 0.16% top
1917 www 1 20 0 30920K 10268K kqread 6 0:01 0.03% nginx
93 root 1 20 0 9164K 5548K select 7 0:04 0.02% devd
5597 root 8 20 0 40284K 17892K select 6 1:25 0.00% rrdcached
1323 root 1 20 0 12484K 12580K select 2 0:02 0.00% ntpd
1522 root 1 20 0 9012K 5024K select 5 0:01 0.00% zfsd
1433 root 1 20 0 127M 105M kqread 2 0:03 0.00% uwsgi-3.7
5622 root 11 20 0 354M 261M nanslp 7 6:57 0.00% collectd
zpools are below:
root@freenas[~]# zpool list -v
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
freenas-boot 110G 1.25G 109G - - 0% 1% 1.00x ONLINE -
da0p2 110G 1.25G 109G - - 0% 1%
pool1 32.5T 2.60T 29.9T - - 1% 7% 1.00x ONLINE /mnt
raidz2 32.5T 2.60T 29.9T - - 1% 7%
gptid/e7aba92d-dc27-11ea-b1e2-00155d450302 - - - - - - -
gptid/ea0339c1-dc27-11ea-b1e2-00155d450302 - - - - - - -
gptid/eadd3d73-dc27-11ea-b1e2-00155d450302 - - - - - - -
gptid/eb18ed64-dc27-11ea-b1e2-00155d450302 - - - - - - -
gptid/eb80f3cc-dc27-11ea-b1e2-00155d450302 - - - - - - -
gptid/eb99fe09-dc27-11ea-b1e2-00155d450302 - - - - - - -
using root@freenas[~]# ./solnet-array-test-v2.sh i can see (the write is showing will take 467 minutes, i am sure write speeds are OK as initially they are fast):
Disk Disk Size MB/sec %ofAvg
------- ---------- ------ ------
da0 130048MB 658 268 ++FAST++
da1 5723166MB 204 83 --SLOW--
da2 5723166MB 176 72 --SLOW--
da3 5723166MB 164 67 --SLOW--
da4 5723166MB 161 66 --SLOW--
da5 5723166MB 186 76 --SLOW--
da6 5723166MB 167 68 --SLOW--
Performing initial parallel array read
Wed Aug 12 04:06:35 PDT 2020
The disk da0 appears to be 130048 MB.
Disk is reading at about 1075 MB/sec
This suggests that this pass may take around 2 minutes
Serial Parall % of
Disk Disk Size MB/sec MB/sec Serial
------- ---------- ------ ------ ------
da0 130048MB 658 0 0 --SLOW--
da1 5723166MB 204 204 100
da2 5723166MB 176 164 93
da3 5723166MB 164 166 101
da4 5723166MB 161 159 99
To add, Windows Storage Spaces with the same hardware is @ 150MB/s constant (pretty much what I am seeing above with the test), but without the slowdown after 1 minute to 30-50MB/sec that i am seeing in actual use of Freenas.
Any other diagnostics i would need to run or ideas on what this could be?
