2

everyone,i have a lot of files write to disk per seconds,i want to disable disk cache to improve performance,i google search find a solution:win32 CreateFile method with FILE_FLAG_NO_BUFFERING and How to empty/flush Windows READ disk cache in C#?.

i write a little of code to test whether can worked:

const int FILE_FLAG_NO_BUFFERING = unchecked((int)0x20000000);

[DllImport("KERNEL32", SetLastError = true, CharSet = CharSet.Auto, BestFitMapping = false)]
static extern SafeFileHandle CreateFile(
      String fileName,
      int desiredAccess,
      System.IO.FileShare shareMode,
      IntPtr              securityAttrs,
      System.IO.FileMode  creationDisposition,
      int                 flagsAndAttributes,
      IntPtr              templateFile);

static void Main(string[] args)
{
    var handler = CreateFile(@"d:\temp.bin", (int)FileAccess.Write, FileShare.None,IntPtr.Zero, FileMode.Create, FILE_FLAG_NO_BUFFERING, IntPtr.Zero);
    var stream = new FileStream(handler, FileAccess.Write, BlockSize);//BlockSize=4096
    byte[] array = Encoding.UTF8.GetBytes("hello,world");
    stream.Write(array, 0, array.Length);
    stream.Close();
}

when running this program,the application get exception:IO operation will not work. Most likely the file will become too long or the handle was not opened to support synchronous IO operations

later,i found this article When you create an object with constraints, you have to make sure everybody who uses the object understands those constraints,but i can't fully understand,so i change my code to test:

var stream = new FileStream(handler, FileAccess.Write, 4096);
byte[] ioBuffer = new byte[4096];
byte[] array = Encoding.UTF8.GetBytes("hello,world");
Array.Copy(array, ioBuffer, array.Length);
stream.Write(ioBuffer, 0, ioBuffer.Length);
stream.Close();

it's running ok,but i just want "hello,world" bytes not all.i trying change blocksize to 1 or other integer(not 512 multiple) get same error.i also try win32 WriteFile api also get same error.someone can help me?

Community
  • 1
  • 1
springchun
  • 193
  • 3
  • 13
  • 1
    You don't have to pinvoke, it is available as FileOptions.WriteThrough. Writing this way is much, *much* slower since it has to wait for the disk. – Hans Passant Jan 01 '12 at 15:25
  • Your best best is to use default options (i.e. do not use write through) and write the entire 60-80kb file in a single shot. – David Heffernan Jan 02 '12 at 11:08

2 Answers2

5

CreateFile() function in No Buffering mode imposes strict requirements on what may and what may not be done. Having a buffer of certain size (multiple of device sector size) is one of them.

Now, you can improve file writes in this way only if you use buffering in your code. If you want to write 10 bytes without buffering, then No Buffering mode won't help you.

Eugene Mayevski 'Callback
  • 45,135
  • 8
  • 71
  • 121
  • yes,thanks.if i have a lot of files on per seconds(25 files on per second),a average size of file is 60-80k,may i can do use no buffer? – springchun Jan 01 '12 at 10:28
  • @user902408 In your particular scenario no buffering mode won't help. That mode is fine in, for example, generation of huge XML files where tons of tags and other data is written in small pieces. But for 60-80 kb large files you won't gain a lot. – Eugene Mayevski 'Callback Jan 01 '12 at 14:17
  • @user902408 Depending on what you are going to do with files further, it's possible that virtual file system (FS in a file) will work better as it will provide a buffering level and will work with one large file on the disk, possibly without buffering (just as you want). – Eugene Mayevski 'Callback Jan 01 '12 at 14:18
1

If I understood your requirements correctly, this is what I'd try first:

Create a queue with objects that have the data in memory and the target file on the disk. You start writing the files first just into memory, and then on another thread start going through the queue, opening io-completion port based filestream handles (isAsync=True) - just don't open too many of them as at some point you'll probably start losing perf due to cache trashing etc. You need to profile to see what is optimal amount for your system and ssd's.

After each open, you can use the async filestream methods Begin... to start writing data from memory to the files. the isAsync puts some requirements so this may not be as easy to get working in every corner case as using filestream normally.

Whether there will be any improvement to using another thread to create the files and another to write to them using the async api, that might only be the case if there is a possibility that creating/opening the files would block. SSD's perform various things internally to keep the access to data fast, so when you start doing this sort of extreme performance stuff, there may be pronounced differences between SSD controllers. It's also possible that if the controller drivers aren't well implemented, OS/Windows may start to feel sluggish or freeze. The hardware benchmarks sites do not really stress this particular kind of scenario (eg. create and write x KB into million files asap) and no doubt there's some drivers out there that are slower than others.

Anonymous Coward
  • 61
  • 2