5

My understanding of how memory mapped files work in C# is that every request for data results in a copy. For instance, if you had a large data structure persisted as a file, using a memory mapped file would result in memory for the actual file mapped into RAM, and a copy residing in a GC heap once it was read from the file.

I'm assuming this is because pointers and the GC don't get along well together generally speaking.

So, is there any way around this?

  • Perhaps via some mixed mode C++ that can expose a managed API over the memory mapped data?
  • What about direct pointer manipulation with unsafe C#?

The general problem I'm trying to solve is sharing a large data structure between multiple processes. The data structure is used to answer a small set of "questions" that can be exposed as a simple API (i.e. basically, a highly specialized index of a bunch of other data).

On a side note, doesn't this make the .NET API useless for the "sharing large amounts of data" scenario?

David
  • 63
  • 1
  • 5
  • 1
    http://code.msdn.microsoft.com/windowsdesktop/Inter-process-communication-e96e94e7 – Robert Harvey May 14 '12 at 22:30
  • Do you read the entire file at once into the .NET application? If you're reading portions of the memory-mapped file, as needed, you allocate a lot of small buffers, but this is not as expensive as you might think. It may be possible to avoid making copies, but is the overhead of making copies really a problem? – Dan Bryant May 14 '12 at 23:46
  • I have .NET applications that very effectively "share large amounts of data" in several ways, including memory mapped files. In particular, one suite shares upwards of 8 gigabytes, and the total memory usage of those programs is surprisingly close to 8 gigabytes. – Jim Mischel May 15 '12 at 02:24
  • I would read the entire contents at once. If I make a copy in each process that needs the data structure then I haven't saved anything from use of MMFs, might as well use a filestream. – David May 15 '12 at 02:35

1 Answers1

4

You can use unsafe code to directly access the mapped memory. I suggest you look into "blittable structs" which are struct types which can be copied around in memory without modification. Here is an example:

struct MyDataRecord { public int X, Y; }

...

for (var i = 0 .. 10) {
 ((MyDataRecord*)pointerToUnmanagedMemory)[i] = new MyDataRecord() { X = i, Y = i * i };
}

This is very performant and kind of convenient.

usr
  • 168,620
  • 35
  • 240
  • 369
  • Hmm, you skipped the named mutex you need to ensure that a process doesn't read data that's being written. That has a knack for throwing "convenient" out of the window. And kill perf. – Hans Passant May 15 '12 at 00:25
  • Looks like unsafe code and blittable structs will be the way to go... Thanks @usr. I also found this after including "unsafe" in my search: http://stackoverflow.com/questions/7956167/how-can-i-quickly-read-bytes-from-a-memory-mapped-file-in-net – David May 15 '12 at 02:29