3

i am working on a C# application and this application is facing memory crunch, because many objects are getting memory allocation in the Large object Heap.

My C# application has to work on many large file (as an string object) and therefore the memory for this string type object is getting allocated again and again from the Large object heap (thus leading to LOH fragmentation).

Since string is an immutable object, a new memory in LOH is always assigned to this object. My question is, Is there a way, I can pre-allocate some memory in the large object heap and always allocate the same memory to the string object.

Here is the thing in more detail: As I mentioned, I am doing processing on these large files. To do the processing , I have to convert it into the string. Even if I use stringBuilder, it won't be much help, because as soon as I convert it into the String a separate memory for this get allocated in the LOH.

So, I was expecting to allocate a bunch of let's say 100 KB in the memory and whenever I read a new file and convert it into string, these 100 KB gets allocated.

Kunal
  • 151
  • 1
  • 2
  • 6
  • 2
    Consider the use of `StringBuilder` which is basically a mutable string. – Lasse Espeholt Jun 11 '11 at 13:06
  • 5
    why are you allocating the same string over and over again? If you've created it once then just pass it to all consumers. – Andrei Jun 11 '11 at 13:06
  • 2
    Why do you think LOH fragmentation is your problem? – svick Jun 11 '11 at 13:12
  • 1
    Why is your code reading all file content in memory at once? Can't you read the file in smaller chunks? – Dan Jun 11 '11 at 14:28
  • It's not the same string, since i am processing different file, the content would be different. – Kunal Jun 12 '11 at 11:46
  • @Kunal: what operations are you doing on the string? Why do you need to convert the StringBuilder to a string? – Andrei Jun 12 '11 at 11:53
  • Re the Edit: 100kB blocks times 3 files is not big, don't solve problems you don't have. – H H Jun 14 '11 at 22:29

3 Answers3

3

You should use the StringBuilder object to work with strings, as it is a mutable string and provide many methods to work with it.

You are saying that every strings are created again and again. If, for some reason, you must use the strings, just Intern them - they will stored once, and will not be added on memory heap again

VMAtm
  • 27,943
  • 17
  • 79
  • 125
1

have you considered using a memory mapped file? http://msdn.microsoft.com/en-us/library/dd997372.aspx

i dont know what you do with the strings, but is there a reason not to work with streams?

NickD
  • 2,672
  • 7
  • 37
  • 58
1

It's unclear what kind of actions you want to do with the strings, your best shot is an estimate of the final result(s). Use a pessimistic (large) estimate and maybe add a margin.

Then create a StringBuilder with the capacity parameter:

var buffer = new StringBuilder(LargestExpectedSize);

This essentially is pre-allocation on the LOH. Avoid (auto-)growing of StringBuilders or List<>s

H H
  • 263,252
  • 30
  • 330
  • 514