4

Because of science reasons I'm solving nine men's morris game. I'm trying to save information about some subset of games states. Unfortunately I reached my memory limit.

I have few very huge array. When new elements came I need to extend these arrays. I have sufficient memory to store these arrays, but I can't afford creating extended arrays where original array is still in memory.

I thought about saving original array to some file, removing it from memory, creating extended array and loading data to it from file.

Is there any fast way to save array size N and load it as first N elements to longer array? To this moment I was using BinaryFormatter, but I don't know if I can use it here.

How can I tell GC to remove original array from memory?

Ari
  • 3,101
  • 2
  • 27
  • 49

1 Answers1

3

Have you investigated the possibility to always store your arrays in a file and work with that?

In C# (as well as Win API) now you can do that with Memory-Mapped Files. Basically this allows you to map in a memory buffer accessible to your app sections of very large files and read/write to them. There's a clear example at the link I've posted, which shows the basics and how you can "navigate" with a in-memory view through a very large file.

In your case you want to define your own file format (nothing fancy) and decide how you separate those arrays in the file. Probably you will want a header with metadata (how many arrays, their offsets, etc) and a contents section divided in multiple segments the very least.

I think this is a more straightforward way of dealing with this than swapping things in/out of memory + having to worry about GC at every step.

Marcel N.
  • 13,726
  • 5
  • 47
  • 72
  • I have this array sorted in order to binary search them. I don't think it would be fast enough using file. – Ari Jul 29 '14 at 20:30
  • @Ari: That's unfortunate. You could still attempt a Fibonacci search, which has a slight performance gain over binary search when the search data is stored on disk. – Marcel N. Jul 29 '14 at 20:34