1

We have an application server with 2 Fusion-IO SSDs put into one volume group forming one single volume. Inside this volume there are 128 files of 40GB each accessed through memory map.

When looking into the iostat, we found the workload not evenly distributed over both disks. The difference is 25%.

What could be the possible causes of this problem? How should I investigate such a problem?

>lsblk
NAME                      MAJ:MIN RM   SIZE RO MOUNTPOINT
...
fioa                      253:0    0   2.9T  0
└─instvg-instant (dm-17)  252:17   0   5.8T  0 /instant
fiob                      253:16   0   2.9T  0
└─instvg-instant (dm-17)  252:17   0   5.8T  0 /instant

>iostat -xk -d fioa -d fiob
Linux 3.0.101-0.47.52-default (...)        08/10/2015      _x86_64_

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
fioa              0.00     0.00 1295.05 3116.02 65326.45 16359.80    37.04     0.29    0.07   0.01   3.82
fiob              0.00     0.00 1847.35 4090.05 88087.96 21608.44    36.95     0.42    0.02   0.01   5.98
Stan
  • 11
  • 1

1 Answers1

0

I'd need the output of lvs -o +devices,segtype to confirm this, but my best guess is that you've just splatted the LV linearly across both PVs. This is a bad idea from a load-balancing perspective, because the filesystem will tend to fill up from start to finish, and the first half of the filesystem is on one PV, and the second half is on the other PV. So, until the filesystem is half full, pretty much all I/O is going to go to the first PV.

To fix this, the LV needs to be created as a striped volume, rather than linear. This can be done with -i 2 passed to lvcreate. Converting an LV to striped is possible, but only if you've got a while pile of spare space, which you don't appear to have here. What this will do is alternate chunks of the LV onto both PVs, so that (for example) the first 8KiB of data will be on the first PV, the next 8KiB on the second PV, the next 8KiB back on the first PV, and so on.

That being said, your I/O profile isn't terribly out of balance. You're never going to get identical I/O on both PVs, simply because the requests going to the PVs aren't going to be perfectly balanced.

womble
  • 96,255
  • 29
  • 175
  • 230