3

My application read lot of data from text files that get into large object heap which eventually cause fragmentation issue. Are there any alternative design approaches that can help to not get these to LOH? I know about StringBuilder but it seem like reading from text file still internally create large strings into LOH.

whoami
  • 1,689
  • 3
  • 22
  • 45
  • LOH only kicks in at 85,000 bytes... I guess you're reading entire files into strings? You'll just have to read it in chunks. – Cory Nelson Mar 16 '15 at 17:54
  • 1
    How are you reading from the file? If that is the problem, then you need to access the file in small chunks, rather than something like `ReadAllLines`. – vcsjones Mar 16 '15 at 17:54
  • Most of places I am doing via ReadLine although there are few places where its done via ReadAllLines as well. When doing it via ReadLine, I still need to combine all those lines as one, which is done using StringBuilder – whoami Mar 16 '15 at 18:30
  • Then it will end up in the LOH. If you can't write smarter code then simply specify a 64-bit operating system for your application, there are no reasons left to not expect one. – Hans Passant Mar 16 '15 at 18:34
  • @HansPassant Can I have some type of memory buffer and somehow have these allocations go from that buffer and not from all over the memory? – whoami Mar 16 '15 at 18:52

1 Answers1

2

If you don't want to change the calls to ReadAllLines there is no way to avoid allocations to the LOH when strings are large enough (short of cloning the .Net github repo, changing the allocation strategy, and recompiling it). Any other solution would see either reading smaller strings from the file, or reading the file as bytes into your own buffer where you then manage allocation and arrangement of the bytes on your own, likely augmented by creating your own string class (since System.String always copies into its own buffer).

However you do now have a choice with .Net >=4.5.1: LOH compaction. To perform this manually, execute the following:

GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce;
GC.Collect();
codekaizen
  • 26,990
  • 7
  • 84
  • 140
  • I believe that this can help improve the fragmentation issue but once LOH is used, there are still other issues as well :) such as GC works more slowly, consumes more CPU and reduces the processing throughput. Since `asp.net core 3.x`, this compaction is done automatically, but I doubt that the other mentioned issues can still affect the performance. – Hopeless Mar 13 '21 at 22:30