2

We are currently investigating high disk latency on a Windows server 2012 r2 that run as an SQL server. It is a virtual machine under VMware and the datastore of the faulty disk is linked with a very high performance LUN on a SAN.

The SAN shows very good response time for the LUN even during incidents and during my test. The datastore also shows very good response time at every moment. The cpu and memory aren't the bottleneck, I have double checked.

Microsoft suggested we use diskspd to test our disk performance. Here is the result of 2 sets of test. I have runned them a couple time, for longer interval and at different time so I am sure the result aren't incidental.

Command Line: diskspd -b64k -o32 -t4 -d60 -w50 -Sw -r -L -c20G -Z1G C:\iotest.data

Total IO thread | bytes | I/Os | MB/s | I/O per s | AvgLat | LatStdDev | file

12623020032 | 192612 | 200.59 | 3209.46 | 38.636 | 21.687

Command Line: diskspd -b64k -o32 -t4 -d60 -w50 -Su -r -L -c20G -Z1G C:\iotest.data

Total IO thread | bytes | I/Os | MB/s | I/O per s | AvgLat | LatStdDev | file

78517239808 | 1198078 | 1247.71 | 19963.34 | 6.410 | 8.202

According to the documentation about diskspd -Sw disable write-though IO and -Su disable software caching. Let me specify that with or without -Sw, on the first command line, the result are the same letting me know that this flag does not have much impact. From this tool(created and managed by the Windows team), we could conclude the cache (disable with -Su) is ruining disk performance but this does not seem right.

My questions are :

  • Why would software caching lower the performance ?

  • Does it impact running application the same way it impacts this test ?

  • IOMeter gives me the same performance as the test without software caching, why ?

Thanks,

Alex
  • 21
  • 4

2 Answers2

0

You are changing two variables, buffered and write through, so you cannot tell what changed. A better comparison would be between -Sb "buffered" and -Su "unbuffered".

Actually, it is quite likely that -Sw is killing your performance. Any storage array worth being called such has write cache, so writes can be acknowledged faster than than they can be committed to the disks in the array. Write through can bypass that.

Keeping checking all available components on the array, SAN and host, software and hardware.

John Mahowald
  • 32,050
  • 2
  • 19
  • 34
  • Thanks for your answer but : Let me specify that without -Sw the result are the same. I can run diskspd -b64k -o32 -t4 -d60 -w50 -r -L -c20G -Z1G C:\iotest.data with our without -sw and optain de same result somewhat proving the little impact of this flag. – Alex Dec 19 '16 at 14:29
  • I edited my question to make avoid confusion. – Alex Dec 19 '16 at 14:35
0

You misunderstand what -Sw does.

As stated by diskspd docs, -Sw actually bypasses OS write cache, as it open the destination file with FILE_FLAG_WRITE_THROUGH.

What surprise me is that your SAN is honoring this flag at all: having a powerloss-protected write cache, SANs and storage arrays does not generally honor the writethrough flag.

On the other hand, the unbuffered and not-writethrough test shows much improved results.

Maybe this is your real problem: your SAN is honoring a (probably useless, has it surely has a persistent cache) writethrough flag.

shodanshok
  • 47,711
  • 7
  • 111
  • 180
  • Thanks for your answer but : Let me specify that without -Sw the result are the same. I can run diskspd -b64k -o32 -t4 -d60 -w50 -r -L -c20G -Z1G C:\iotest.data with our without -sw and optain de same result somewhat proving the little impact of this flag. – Alex Dec 19 '16 at 14:29
  • I will edit my question to make it more clear that with or without -sw the result are the same. – Alex Dec 19 '16 at 14:30