Uneven disc throughput in RAID1 configuration

Question

I am having performance issue caused by slow IO in server which has two identical SSDs in RAID1 configuration. By using "atop" noticed that discs usage is no even:

DD |           md2 |  busy      0%  | read       0  |  write   1717 |  MBr/s   0.00 |  MBw/s   0.58  | avio 0.00 ms  |
DSK |           sdb |  busy     99%  | read       0  |  write    842 |  MBr/s   0.00 |  MBw/s   0.58  | avio 11.8 ms  |
DSK |           sda |  busy     11%  | read       0  |  write   1058 |  MBr/s   0.00 |  MBw/s   0.58  | avio 1.01 ms 

Question is what could be cause of it? Why sdb usage is much higher? I already noticed same issue on few servers so it is very unlikely that all of them have faulty sdb. Also, checked discs info by using hdparm to make sure they are identical. Also, it only happens on servers running production MySQL server. I was trying to reproduce this issue by simply writing and reading from partition, but I was not able to reproduce same results this way. Thank you for suggestions.

[root@CentOS-67-64-minimal ~]# cat /proc/mdstat 
Personalities : [raid1] 
md2 : active raid1 sdb3[1] sda3[0]
  232753344 blocks super 1.0 [2/2] [UU]

md1 : active raid1 sdb2[1] sda2[0]
  524224 blocks super 1.0 [2/2] [UU]

md0 : active raid1 sdb1[1] sda1[0]
  16777088 blocks super 1.0 [2/2] [UU]

unused devices: <none>

[root@CentOS-67-64-minimal ~]# hdparm -tT /dev/sda

/dev/sda:
Timing cached reads:   28484 MB in  2.00 seconds = 14263.62 MB/sec
Timing buffered disk reads: 1096 MB in  3.00 seconds = 365.15 MB/sec
[root@CentOS-67-64-minimal ~]# hdparm -tT /dev/sdb

/dev/sdb:
Timing cached reads:   21656 MB in  2.00 seconds = 10841.67 MB/sec
Timing buffered disk reads:  14 MB in  3.95 seconds =   3.54 MB/sec

[root@CentOS-67-64-minimal ~]# iostat -x 1
Linux 2.6.32-573.3.1.el6.x86_64 (CentOS-67-64-minimal)  2015.11.20      _x86_64_        (8 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
       8,96    0,13    4,22    3,93    0,00   82,75

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sdb               0,06   383,28    0,22  164,10    20,26  4275,72    26,15     2,26   13,76   3,21  52,79
sda               2,01   329,12    1,50  218,25   168,39  4275,72    20,22     0,17    0,77   0,35   7,71
md0               0,00     0,00    0,00    0,00     0,01     0,00     8,00     0,00    0,00   0,00   0,00
md1               0,00     0,00    0,00    0,00     0,01     0,00     7,62     0,00    0,00   0,00   0,00
md2               0,00     0,00    1,75  546,03   172,42  4274,38     8,12     0,00    0,00   0,00   0,00

---

MadHatter edits:

Here's my iostat output under very light load; note the %util on sdb's spindle (you can distinguish my output from Nerijus' by the different hostname in the prompt, and I'll keep my edits below the (above) line):

[me@lory ~]$ iostat -x 1
[...]
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.25    0.00    0.50    0.00    0.00   99.25

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sdb               0.00     0.00    0.00    4.00     0.00    20.00     5.00     1.00  272.50 250.00 100.00
sda               0.00     0.00    0.00    4.00     0.00    20.00     5.00     0.07   17.75  17.75   7.10
md1               0.00     0.00    0.00    5.00     0.00    40.00     8.00     0.00    0.00   0.00   0.00
dm-0              0.00     0.00    0.00    4.00     0.00    32.00     8.00     2.51  272.50 250.00 100.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
md0               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

And here's my hdparm output:

[me@lory ~]$ sudo hdparm -tT /dev/sda
One-time password (OATH) for `me': 

/dev/sda:
 Timing cached reads:   1730 MB in  2.00 seconds = 864.60 MB/sec
 Timing buffered disk reads: 436 MB in  3.00 seconds = 145.12 MB/sec
[me@lory ~]$ sudo hdparm -tT /dev/sdb

/dev/sdb:
 Timing cached reads:   1580 MB in  2.00 seconds = 789.63 MB/sec
 Timing buffered disk reads:  14 MB in  8.43 seconds =   1.66 MB/sec

I can also confirm that my /proc/mdstat indicates no resyncing, and that stopping mysqld doesn't make the problem go away.

Answer 1

Possible reasons i see:

1. Because of disk (+ maybe controller) caches - they're never working in same way
2. Different firmware
3. If you don't use TRIM on all levels, then SSD speed will go lower, because from the view of the controller (inside the disk), disk will be full, then SSD disks are slower. Benchmarks show, usually, disk speed depends on its usage too.
4. Related to previous - allocation place matters, is it SSD or not, some/most SSD's have RAID-like structure, therefore maximal speed depends on where do you write data
5. Different HW from disk to bus (including cable, link speed, controller etc)
6. OS block cache for the disk - maybe memory is not enough to balance cache for both disks...

hdparm -tT not gonna give you real results anyway, it's buffered and not always as transparent as we expect, it doesn't necessary expose how disks work internally, especially in 2 secs.

Probably, they're fine.

Answer 2

We have figured out that your SBD it's not in health, this is my answer. Take a look , you need to obtain a value like this (tested on 3 different server in production with high I/O)

1-Timing buffered disk reads: 560 MB in  3.00 seconds = 186.43 MB/sec
2-Timing buffered disk reads: 276 MB in  3.09 seconds =  89.23 MB/sec
3-Timing buffered disk reads: 326 MB in  3.00 seconds = 108.66 MB/sec
5-Timing buffered disk reads: 528 MB in  3.00 seconds = 175.97 MB/sec
6-Timing buffered disk reads: 528 MB in  3.00 seconds = 175.94 MB/sec

So your 1.66 MB it's so far.. then produce overload on your system. I hope it helps!