0

As the Dell forums are a veritable disaster I'm trying here in case somebody has some hints for me here :)

We have a system with 3 drives:

  1. SSD for the OS
  2. SSD containing a ZFS pool backing LXD containers
  3. A "backup" storage HDD drive

The system is running Ubuntu Bionic.

Now these area all connected to the PERC controller in a non-raid configuration. All firmware is up to date on the server.

What we are seeing is when we stream data to the HDD drive using rsync or ftp from remote network storage, the write operations to the HDD drive slows down the entire system

It is as if high IOwait on the HDD drive blocks IO operations on the SSD drives. Which to me is highly counter-intuitive.

Why would write operations to the HDD drive cause the entire system to lock up and slow down access times on the SSD drives? Is the controller at fault, or are there some knobs in the Linux Kernel which can be turned here?

Did I shoot myself in the foot by mixing in an HDD drive with my SSD drives? :)

Arni J
  • 65
  • 6
  • 1
    The H310 card sucks rocks, and is at the heart of your problem. See https://www.dell.com/community/PowerEdge-HDD-SCSI-RAID/Terrible-performance-with-PERC-H310-mini/td-p/5079896 - Using SSDs mitigates this pain, but really the card is not of merchantable quality and needs to be replaced with something better. – davidgo Nov 03 '19 at 19:14
  • 1
    When in RAID mode, the card itself handles an IO queue, which, when full, affects the entire disk subsystem. If you don't need RAID functionality on your "backup" HDD, you may set it in non-RAID mode which by-pass the IO queue from the RAID card (for this specific disk). But please, note that converting from raid-capable to non-raid will erase data (you backup data). – sfk Nov 04 '19 at 08:26

1 Answers1

0

The comment from @sfk should be the answer on the question.

Shared I/O queue/buffering is absolutely the cause of the problem here.

You can either cross-flash the card to "IT" mode with LSI 9211-8i firmware, or just buy an LSI 9207-8i online (they're very affordable now) if you don't want to flash the PERC. Just be sure to get a full backup of everything on the disks beforehand that you will need to restore from - a "non-RAID" disk on a PERC still has RAID virtual disk configuration, and won't show up normally on a non-RAID card (like this one in IT mode).

Edit: Another option you might consider as a work-around would be finding a way to limit I/Os that get sent to that disk device (heading in the direction of the "knobs" you mentioned turning). I see some ideas/options here such as cgroups and ionice, but not entirely certain which would be a good fit for you. Anything that can cause writes to that disk to only be "issued" at a rate slower than the disk can actually handle would likely be beneficial, so that I/Os aren't queuing up until the HBA's IO queue gets filled to the brim.

JimNim
  • 2,776
  • 13
  • 24
  • Darn. However, I feel like I'm getting slightly conflicting information here. sfk claims that when the card is in RAID mode this issue would crop up - but I just checked and the card is definitely "unconfigured" and all disks show as Non-Raid in the PERC BIOS. So on the surface of it your answer seems to be the more accurate in that PERCs "non-raid" is a "fake" and my options are a) flash the card and pray or b) get a new card – Arni J Nov 04 '19 at 19:53
  • Yeah, I guess slightly conflicting info... Configuring a disk as "non-RAID" MAY actually not place RAID metadata config on a disk as I stated too - I just EXPECT it would. Added more thoughts/ideas in an edit to my answer too, FYI. Finding a way to throttle I/Os for that slower disk might be a more elegant solution (and less of a pain for you). – JimNim Nov 04 '19 at 20:19
  • Great suggestion Jim - I had thought something along the same lines. I think I'll look into something like ionice for this. Thanks for all your input here! – Arni J Nov 04 '19 at 21:36
  • Comment back if ionice ends up doing the trick for you - I’m actually very curious to hear how that works out! – JimNim Nov 04 '19 at 21:37
  • I ended up using the built-in bandwidth limiting of lftp for our synch job instead of ionice, and that worked pretty well in limiting writes to the offending drive. We have ended up with a more permanent solution however, and that is to redesign our workflows so we don't need to use that drive at all anymore. As an aside, I am ordering in a new and better controller however, as I want to test/see if it makes a difference to our SSD throughput in general. I may report my findings back here, i.e. how much better (hopefully) disk I/O gets if you get a good controller. – Arni J Dec 05 '19 at 11:57