1

I just went for some disk performance tests on my DIY NAS and noticed some very poor performace results with my NVME disk writes !

My NAS runs Debian 10 with this config :

  • ASROCK X470D4U with Ryzen 5 3600
  • 16 Gb ECC RAM
  • 1 NVME for OS (ADATA x8200 Pro 256MB). This is a PCIe Gen 3 x4 on a PCIe Gen 2 x2 slot, given for 3500MB/s Read & 3000MB/s Write.
  • 4 SATA HDD in software RAID 6 (WD RED 10TB)
  • 1 SATA HDD (WD Green 2TB)

NVME write speed : 320 MB/s !

# time ( dd if=/dev/zero of=test.img bs=1G count=10 status=progress ; sync )
10737418240 bytes (11 GB, 10 GiB) copied, 27 s, 399 MB/s
10+0 records in
10+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 33.5806 s, 320 MB/s

real    0m34.650s
user    0m0.001s
sys     0m8.943s

NVME read speed : 1,7 GB/s

# /sbin/sysctl -w vm.drop_caches=3; time ( dd if=test.img of=/dev/null bs=10G count=5 status=progress )
vm.drop_caches = 3
2147479552 bytes (2.1 GB, 2.0 GiB) copied, 1 s, 1.6 GB/s
dd: warning: partial read (2147479552 bytes); suggest iflag=fullblock
10737397760 bytes (11 GB, 10 GiB) copied, 6 s, 1.7 GB/s
0+5 records in
0+5 records out
10737397760 bytes (11 GB, 10 GiB) copied, 6.27933 s, 1.7 GB/s

real    0m6.358s
user    0m0.000s
sys     0m4.593s

For comparison, here are the results of the RAID 6 (5400 rpm) filesystem and a single HDD (5400 rpm) :

RAID 6 write speed : 275 MB/s
RAID 6 read speed : 290 MB/s
Single disk write speed : 120 MB/s
Single read speed : 125 MB/s

So, I think that the read speed is somehow normal for the NVME drive as it is plugged on a PCIe x2 slot instead of a x4 slot, so 3500 MB/s / 2 should be near the 1,7 GB/s I've found. But I can't explain the poor performance when writing...

Any idea of what is going on ?

Stéphane

sbocquet
  • 23
  • 5
  • I have a very similar problem. Also limited to ~300Mbs https://forum.openmediavault.org/index.php?thread/48572-slow-speeds-bottleneck-for-10gbe-pcie-nvme-adapter/&postID=356208#post356217 – square_eyes Jul 02 '23 at 01:48

1 Answers1

3

Unfortunately, it's now well known that consumer SSD performance vary wildly and rarely matches expectations. There are several problems going on:

  1. the first batch of a new SSD series (the batch that will be reviewed by Tom's Hardware, Anandtech, ServeTheHome etc) are generally much better and faster than the following batches. Often, after a few months the hardware changes completely (flash type and speed, sometimes even controller) while retaining the same model number!
  2. write speed: there are MLC, TLC and QLC SSDs. TLC and QLC have very low write performance, and use an SLC (single cell) flash cache, generally from 1 to 4 GB in size. The top write speed is only obtained while writing to the SLC cache; any write larger than the cache will slow down dramatically.

You can check the cache effect by running dd with the status=progress and oflag=direct options. You'll see the speed be high for the first 1, 2 or 4 GB (depending upon the cache size), then drop to the TLC or QLC speed afterwards (down to 80-100MB/s on TLC SSDs, or even lower on QLC ones!).

wazoox
  • 6,918
  • 4
  • 31
  • 63
  • 1
    Thanks for those informations. I'm aware of most of them, and that's why I bought this NVMe on the very beginning, almost 2 years ago. After more indeep research, It seems that it could be a temperature throttling problem... At idle, the SSD is at 37°C, and I have around 500 to 600MB/s average writing performance. After 30s, the SSD temperature is at 45 to 50°C, and the writing performance fall to 300 MB/s... I've just ordered a NVMe Heatsink in order to confirm. – sbocquet Mar 18 '22 at 17:22
  • But even at 600MB/s, that is far behind the regular numbers I could have seen (here for example https://www.servethehome.com/adata-xpg-sx8200-pro-1tb-nvme-ssd-review/3/). – sbocquet Mar 18 '22 at 17:23
  • If the SSD isn't new, what filesystem are you using? Notice that ext3 doesn't support TRIM. You may want to force a "TRIM" manually before running the test; see https://kb.plugable.com/data-storage/trim-an-ssd-in-linux – wazoox Mar 18 '22 at 18:19
  • The file system is ext4, and I trim the SSD every night using the fstrim command. – sbocquet Mar 19 '22 at 23:20
  • @sbocquet then you should try as I said, checking the numbers while dd is running. Most probably the drive reaches its theoretical performance only on the first 1 to 4 GB writing... then drops to 50MB/s for the rest. – wazoox Mar 21 '22 at 07:59