We have 5 Toshiba Px04Srb192 SSDs (270K random read IOP as per spec), setup in hardware Raid5. Running fio is giving 250K IOPs which is much below what I was expecting.
/fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=/ae/disk1/test --bs=4k --iodepth=96 --numjobs=1 --size=8g --readwrite=randread
test: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=96
fio-2.0.9
Starting 1 process
Jobs: 1 (f=1): [r] [100.0% done] [961.6M/0K /s] [246K/0 iops] [eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=342604: Tue Feb 12 23:58:01 2019
read : io=8192.0MB, bw=991796KB/s, iops=247948 , runt= 8458msec
cpu : usr=10.88%, sys=87.74%, ctx=437, majf=0, minf=115
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued : total=r=2097152/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=8192.0MB, aggrb=991795KB/s, minb=991795KB/s, maxb=991795KB/s, mint=8458msec, maxt=8458msec
Disk stats (read/write):
sdb: ios=2083688/0, merge=0/0, ticks=265238/0, in_queue=265020, util=98.53
lspci -nn | grep RAID
18:00.0 RAID bus controller [0104]: LSI Logic / Symbios Logic MegaRAID Tri-Mode SAS3508 [1000:0016] (rev 01)
I was expecting 5 SSDs IOPs to be at least 2 times individual SSD. Is that right? Any suggestions why we are seeing low IOPs?