Hybrid RAID (SDD+HDD) gives unexpected results

Question

I am doing some experiments with hybrid RAID in Linux. My test consists of the following:

2x256GB SSD in RAID 0 (/dev/md1)

2x256GB HDD in RAID 0 (/dev/md2)

Then I made md1 and md2 into a RAID 1 (/dev/md127) and marking the slow HDD (md2) as --write-mostly.

Essentially, my goal is to get maximum performance AND disk space out of my SSDs, but at the same time be "safe" from drive failures. I understand that loosing one of the SSDs would mean that I fall back on slow HDDs, but that's a price I am willing to pay compared to loosing all data. Besides, it would only be for a few hours until the broken SSDs gets replaced and RAID repaired.

root@s1 / # cat /proc/mdstat
Personalities : [raid0] [raid1] [linear] [multipath] [raid6] [raid5] [raid4] [raid10]

md2 : active raid0 sdd1[1] sdc1[0]
      498802688 blocks super 1.2 512k chunks

md127 : active raid1 md1[2] md2[1](W)
      498671616 blocks super 1.2 [2/2] [UU]
      bitmap: 1/4 pages [4KB], 65536KB chunk

md1 : active raid0 sdb2[1] sda2[0]
      498802688 blocks super 1.2 512k chunks

Now, running a simple throghput benchmark on the 3 raid devices gives a (for me) surprising results:

root@s1 / # hdparm -t /dev/md1

/dev/md1:
 Timing buffered disk reads: 2612 MB in  3.00 seconds = 870.36 MB/sec
root@s1 / # hdparm -t /dev/md2

/dev/md2:
 Timing buffered disk reads: 812 MB in  3.01 seconds = 270.14 MB/sec
root@s1 / # hdparm -t /dev/md127

/dev/md127:
 Timing buffered disk reads: 1312 MB in  3.00 seconds = 437.33 MB/sec

RAID 0 SSD gives 870 MB/sec

RAID 0 HDD gives 270 MB/sec

RAID 1 HYBRID gives 437 MB/sec.

As the HDD raid has been marked as --write-mostly, I would assume that a pure read test would not touch the HDD at all, so what is going on here? I would assume that the hybrid benchmark would give similar results as the pure RAID 0 SSD.

At a first glance, it looks like the HDD somehow is slowing down the RAID, by being partly used for the read (even though I told it not to do reads on the HDD). However, if I have a file copy running on the HDDs while running the hdparm benchmark, I get the same result! If the HDDs WERE used, I would assume the benchmark would give even slower results if the HDDs were used for other tasks during the benchmark.

I hope some Linux raid expert can shed some light to my problem. Thanks!

Please don't do this, we've had dozens of new users ask us about this kind of thing over the past decade, it's never worth the effort - also it's not in production so it's not appropriate for this site anyway. — Chopper3, May 21 '19 at 17:59
Not worth the effort? According to who? When doing a fio benchmark with 4kb random reads on the my hybrid RAID, I get 180k IOPS! Doing the same on just the HDD RAID, I get under 1k. Definitely worth it for a database server. — Daniele Testa, May 22 '19 at 10:53
There is no reason to mark this question off-topic. Today softraids are preferred over hardware raid because you can rebuild them on any hardware. For people considering hybrid softraids, this question and its answers can be helpful. Before something goes into production you consider a technology, try it and deploy it. "Not in production" is in this case not a reason to mark the question off-topic. — Adrian Zaugg, Jul 15 '20 at 08:17

score 4 · Answer 1 · answered May 21 '19 at 17:57

I would assume that a pure read test would not touch the HDD at all, so what is going on here?

This is an incorrect assumption. In particular look at the read_balance() function in drivers/md/raid1.c.

 /*
   * If buffered sequential IO size exceeds optimal
   * iosize, check if there is idle disk. If yes, choose
   * the idle disk. read_balance could already choose an
   * idle disk before noticing it's a sequential IO in
   * this disk. This doesn't matter because this disk
   * will idle, next time it will be utilized after the
   * first disk has IO size exceeds optimal iosize. In
   * this way, iosize of the first disk will be optimal
   * iosize at least. iosize of the second disk might be
   * small, but not a big deal since when the second disk
   * starts IO, the first disk is likely still busy.
   */

  /*
   * If all disks are rotational, choose the closest disk. If any disk is
   * non-rotational, choose the disk with less pending request even the
   * disk is rotational, which might/might not be optimal for raids with
   * mixed ratation/non-rotational disks depending on workload.
   */

(Waves hand) If the ssd is "overloaded" then the idle hdd will be selected. /sys/block/X/queue/optimal_io_size and monitoring the number of pending requests may be helpful to determine what is happening.

If you want deep understanding you're going to have to understand read_balance().

What you are writing makes sense, BUT, why do I then get the same speed if I do a file copy on the HDD at the same time as I do the benchmark? 1. It should then see that the HDD is NOT idle and therefor not use it (or use it less) and the benchmark numbers should increase. 2. If it still used the HDD as much, the result should decrease, as the HDD part will get slower. But now, the numbers shows the exact same result. How is that possible? — Daniele Testa, May 22 '19 at 06:39
I would like to add that doing a benchmark directly on the HDD RAID 0 while copying a file shows a dramatic decrease in performance. So still, why are the benchmark numbers constant when benchmarking the hybrid RAID 1? — Daniele Testa, May 22 '19 at 07:34

Hybrid RAID (SDD+HDD) gives unexpected results

1 Answers1