really low iops on ssd raid (lower than 1000)

Question

I'm having serious performances problems on a server, with really low iops.

server :

ProLiant DL360p Gen8
with HP Smart Array P420i Controller (firmware v6)
with 4x Samsung 840 - 512GB on raid 1+0

here's one samsung 840 details :

hpacucli ctrl slot=0 pd 2I:1:6 show detail
physicaldrive 2I:1:6
Port: 2I
Box: 1
Bay: 6
Status: OK
Drive Type: Data Drive
Interface Type: Solid State SATA
Size: 512.1 GB
Firmware Revision: DXM04B0Q
Serial Number: S12SNEAD102607E
Model: ATA     Samsung SSD 840
SATA NCQ Capable: True
SATA NCQ Enabled: True
Current Temperature (C): 16
Maximum Temperature (C): 70
Usage remaining: 100.00%
Power On Hours: 0
SSD Smart Trip Wearout: False
PHY Count: 1
PHY Transfer Rate: 6.0Gbps
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6

and here's the benchmark result

sudo fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
    test: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=64
    fio-2.1.3
    Starting 1 process
    test: Laying out IO file(s) (1 file(s) / 4096MB)
    Jobs: 1 (f=1): [m] [99.8% done] [7152KB/2400KB/0KB /s] [1788/600/0 iops] [eta 00m:02s]
    test: (groupid=0, jobs=1): err= 0: pid=36718: Thu Mar  5 18:15:12 2015
    read : io=3071.7MB, bw=2536.5KB/s, iops=634, runt=1240097msec
    write: io=1024.4MB, bw=866133B/s, iops=211, runt=1240097msec
    cpu          : usr=0.28%, sys=1.18%, ctx=401767, majf=0, minf=2347
    IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
    submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
    issued    : total=r=786347/w=262229/d=0, short=r=0/w=0/d=0
    Run status group 0 (all jobs):
    READ: io=3071.7MB, aggrb=2536KB/s, minb=2536KB/s, maxb=2536KB/s, mint=1240097msec, maxt=1240097msec
    WRITE: io=1024.4MB, aggrb=845KB/s, minb=845KB/s, maxb=845KB/s, mint=1240097msec, maxt=1240097msec
    Disk stats (read/write):
    dm-1: ios=785968/267543, merge=0/0, ticks=50476776/28341176, in_queue=79704028, util=100.00%, aggrios=788659/265218, aggrmerge=1953/2589, aggrticks=50709304/27483028, aggrin_queue=78191444, aggrutil=100.00%
    sda: ios=788659/265218, merge=1953/2589, ticks=50709304/27483028, in_queue=78191444, util=100.00%

running the same bench on normal hdd shows up more iops than on the ssd

Any idea ?

I wouldn't know... because consumer SSDs don't really belong in these servers... But can you post some details on the filesystem layout and type? — ewwhite, Mar 05 '15 at 19:46
@Charly - Not tempted to use a supported configuration? What are you planning on using this for? It better not be doing any writes — Chopper3, Mar 05 '15 at 19:49
I'm seeing a few questions lately about poor performance with HP servers and Samsung SSDs. I can speak from experience that Samsung 840 and 850's perform exceptionally well in Dell servers (with LSI controllers). Maybe there's some issue with the HP controllers. — Mark Henderson, Mar 05 '15 at 20:29
Have you read the following? http://h20564.www2.hp.com/hpsc/doc/public/display?docId=emr_na-c03015037 — pauska, Mar 05 '15 at 20:30
@MarkHenderson We've ALWAYS known that HP controllers when used with HP disks are fast and reliable and that the opposite is true when used with non-HP disks. — Chopper3, Mar 06 '15 at 09:09
we've had this server since september 2013, and it's started to be slow recently. I'm not sure the 840 pro existed in 512GB at this date. Filesystem is a basic ext4 with 886GB on / — Cactusbone, Mar 06 '15 at 09:53

score 2 · Accepted Answer · answered Mar 05 '15 at 21:26

Due to how MLC SSDs work, they need a decently-sized local DRAM cache to absorb incoming writes while simultaneously write to the backing NAND.

However, hardware RAID cards often disable the disk's cache and exclusively rely on own (on-card) DRAM cache. While this is not a problem with classic HDD, the intrinsic not-overwriting nature of NAND chips (and SSDs build over them) cause a massive performance loss if the internal DRAM cache can not be used.

To give an example: a 256 GB Crucial M550, with sequential write of over 400 MB/s, drops to 5 MB/s with internal cache disabled. With random write the loss is equally severe. This is the precise reason why enterprise SSD used (in the past) SLC NANDs: they program time is way lower compared to MLC or even eMLC.

For this reason, I am always a little nervous when combining consumer-class SSDs with branded hardware RAID cards. On the other side, Linux mdraid works fantastically well with SSDs of any class.

For you specific case, you can do some test trying to re-enable the internal disk cache (DELL/LSI controller give this possibility, I don't know about HP controller). However, remember that with disk's write-cache enabled you are exposed to data loss in the event of a power outage (unless your disks have power-loss protection, but this is quite rare in the consumer space).

Thanks, I'll try tuning the controller. (or use mdraid if i can get the controller to give me the disks as-is) — Cactusbone, Mar 06 '15 at 11:03
SmartArrays are basically LSI gear with custom firmware. Of course you [can enable the drives' write cache](http://h30499.www3.hp.com/t5/ProLiant-Servers-ML-DL-SL/How-to-activate-write-cache-on-P400-Smart-Controller/td-p/4732601). — the-wabbit, Mar 20 '15 at 20:55
activating the drive write cache did indeed boost my iops by a factor 10 ! `sudo hpacucli controller slot=0 modify drivewritecache=enable` — Cactusbone, Aug 07 '15 at 14:18

really low iops on ssd raid (lower than 1000)

1 Answers1