4

We are having an issue with one of our servers. When we copy bigger files (large means 50MB and larger in that case), the copy operation (C:\ to C:) starts normally, but then starts to lag, going down to 100kb/s and making the whole server hang (our application can't return results from the SQL Server anymore, so the application hangs for users).

The Intel RST shows all green on SMART. Here are the system specs:

  • Server: HPE ML10
  • Storage: 3x HP 3TB in RAID5 configuration
  • OS: Windows Server 2012 R2
  • Server Roles: Domain Controller, Application Server (SQL Server and .NET app)
  • Storage Settings: Stripe size: 128KB, Write-cache buffer flushing: Enabled, Cache-Mode: Off, Size of physical and logic sectors: 512 bytes

I'm not a server expert, so I'm not sure if I have those things set up correctly. What could be the problem here?

EDIT: I'm not an expert on these things (developer). So maybe I'm doing something simple wrong.

EDIT2: https://i.stack.imgur.com/3FyMU.jpg Disk write performance is extremely poor. But no total hangs like when I copy with Windows Explorer. I guess the hanging explorer jams the message pump, clogging the system. Could a migration to a RAID1/10 fix the issue in your opinions?

LueTm
  • 101
  • 1
  • 8
  • 4
    3TB disks in R5 - please don't do that, you must really not like your data - R5 is dead as a tool and has been for about a decade - can bore you with the maths but it *will* kill your data at some point, R1/10 and R6/60 are the only games in town. – Chopper3 Sep 07 '17 at 08:03
  • How are you copying this? Windows Explorer? Have you tried a tool like `robocopy`? – duenni Sep 07 '17 at 08:26
  • For now Windows Explorer, yes. We have third party software that copies things onto the server. – LueTm Sep 07 '17 at 09:28
  • Can you check the drive and RAID logs for errors? Have you defragged the filesystem? – ptman Sep 07 '17 at 11:35
  • Obviously your RAID is stalling on write and blocking all IOs. Maybe a failing drive. Any RAID array log? What's the RAID controller? are the drives enterprise drives or desktop drives? What's the SMART status of the individual drives? – wazoox Sep 07 '17 at 11:35
  • The Intel RST Tools finds no issues with the disk (status is 'normal'), wmic says all OK. They are enterprise drives. I'm googling how to get to the RAID array log – LueTm Sep 07 '17 at 11:58
  • Accessing the RAID logs seems to be hard. I have to install web services and stuff. stocli can't find any compatible controllers. Can I migrate this installation onto a RAID1 or RAID10? (e.g. from a Veeam backup)? – LueTm Sep 07 '17 at 13:03
  • See edit2, please! – LueTm Sep 07 '17 at 18:42
  • I can bet at least one of your disks is dying... – BaronSamedi1958 Sep 07 '17 at 19:36

1 Answers1

2

If I interpret "Cache-Mode: Off" correctly it's completely understandable that write performance sucks. Check out whether copying/reading from the RAID (to network or NUL) is I problem or copying/writing to the RAID - I my guess is correct, only writing to the RAID is a pain.

RAID5 is distributed - each stripe consists of (in your case) three segments: data1, data2 and parity12. Now, when some data is written to the array it can't just be written to a data segment because the parity wouldn't match any more.

If data1 is written to/changed the controller needs to either:

  1. read data2, recalculate parity12, write data1, write parity12 (for small arrays)
  2. read old data1, read parity12, remove old data1 parity from parity12, recalculate parity12 with new data1, write data1, write parity12 (for larger arrays)

So, whenever there's a change the controller operations are amplified by three! If these cannot be cached, each write will result in three operations to be performed and your application needs to wait. With cache, many read and write operations can be omitted and the performance hit will be much less.

The only exception to this write operation amplification is when you write a whole stripe at once: just take data1 and data2 from buffer, calculate parity12 and write all three segments. That's an amplification by just 1.5. However, for being able to combine all incoming data into full stripes you need to be able to queue the data. Guess what, you need cache again.

In a nutshell: if you use RAID5 or RAID6 you absolutely require cache - it's not a luxury. Too little or even no cache at all will kill your performance. If it's a software or hosted RAID with configurable cache, set aside at least 512 MB, better 1 or 2 GB and it'll "fly". RAID5 with three drives will be no performance wonder but it can work fine.

Edit: the HP ML10 G9 has a chipset-integrated Intel RST SATA RAID controller - host RAID. Depending on which exact model and controller is used, cache should be configurable somewhere.

Zac67
  • 10,320
  • 2
  • 12
  • 32
  • You have a point but RAID5 has 4x write penalty so 100 KB/sec is too slow even for RAID5 **working properly**. – BaronSamedi1958 Sep 07 '17 at 19:37
  • Don't forget the drives work mechanically - non-consolidated reads/writes can be severely penalized by access latency. I just noticed the OP's edit2 which pretty much confirms my theory. – Zac67 Sep 07 '17 at 19:41
  • Thank you! Makes sense. So we need to make absolutely sure that the UPS shuts down the server on a power outage to prevent data loss with write cache enabled, correct? And apparently also trick the Domain Controller to leave that setting alone... Or convert to RAID 1 or 10. – LueTm Sep 08 '17 at 05:40
  • Yes, pretty much. I'm not aware of Windows fiddling with storage controller settings, but if that is the problem it needs to be solved. You need a UPS for any serious server anyway and configuring auto shutdown on a physical server can't bee too hard. – Zac67 Sep 08 '17 at 06:27
  • Accepted as answer. We reproduced the issue with the same settings on another machine. One question though, they now disabled "Write cache buffer flushing". This should only be enabled with a redundant power supply that shuts down the server safely, correct? – LueTm Sep 14 '17 at 06:26
  • 1
    Yes - if you like your data. I don't know the exact settings, but a _read cache_ is dearly required for R5. A _write cache_ can improve performance significantly, but a _write back cache_ introduces data risks that need to be mitigated by a UPS. A _write through cache_ doesn't hurt and is pretty much a read cache for currently written data.. – Zac67 Sep 14 '17 at 06:32
  • Thank you for the explanation. We plan to use Write-Through cache as a temporary remedy, and move to RAID10 without cache if this works well in the lab. – LueTm Sep 14 '17 at 06:38