5

My goal is to load 1.48 million items from XML files into a in memory generic collections dictionary using C# ASP.NET MVC along with ability to render filter lists to views. Right now it works with about 1/2 million items in my dictionary as a single user in VS debug mode. Any more I get out of memory errors - right now I get: myDictionary.Count results in System.TypeInitializationException - 'The function evaluation was disabled because of an out of memory exception.'

In the past it has also been mscorlib.dll during the load from XML to Dict that has complained about out of memory.

I have plenty of system memory left, how can I give more to this MVC Web application? BTW, I tried XML to a Perl Dictionary and it works just fine with 1.5 million objects - no problem. I don't wish to write my app in Perl. If Perl can do it, C# has to be able to do it - I just have not been able to find a solution searching the web yet.

Example Code:

Dictionary<string, msg> masterDictionary = new Dictionary<string, mgs>();

foreach (string file in filePath)
{
    XDocument xdoc = XDocument.Load(file);
    Dictionary<string, msg> fileDictionary = xdoc
        .Descendants("msg")
        .ToDictionary(m => m.Element("msgId").Value,
                      m => new msg
                           {
                               msgId = m.Element("msgId").Value,
                               msgType = m.Element("msgType").Value,
                               name = m.Element("name").Value
                           });

    //now insert your new values into the master
    foreach(var newValue in fileDictionary)
        masterDictionary.Add(newValue.Key, newValue.Value);
}
Bill
  • 915
  • 2
  • 13
  • 23
  • 2
    You don't show any code that allows anyone else to know just how much you are trying to load into memory – Gary Walker Oct 30 '13 at 20:30
  • Also the .Net runtime can use a lot of memory -- a few hundred MB in some cases. I know you used to never get more than 2GB is a CLR, even when using 64 bit windows, don't know if this is true in core recent CLR versions. – Gary Walker Oct 30 '13 at 20:33
  • Running the performance Monditory it looks like I have about 10 gigs to spare. – Bill Oct 30 '13 at 20:39
  • @GaryWalker - not clear what your comment "never get more than 2GB" means, but both x86/x64 don't allow one single allocation to be bigger than 2GB (except [arrays](http://stackoverflow.com/questions/13130676/large-array-support-in-asp-net) ) . In 32bit you should be able to use up to 4GB of memory (assuming your OS is x64, but process is x86), on x64 you should not have OOM exceptions due to amount of allocated memory, but only due to size. – Alexei Levenkov Oct 30 '13 at 20:40
  • I believe the asp.net stack runs in a 2 GB processes, and the stack can use a few hundred meg itself. – Gary Walker Oct 30 '13 at 20:41
  • @GaryWalker - IIS runs ASP.Net processes as x64 just fine... – Alexei Levenkov Oct 30 '13 at 20:42
  • Alexi -- A single CLR object cannot exceed 2 GB, e.g. a huge array or stream will always fail. Actually, since it will try to allocate a contiguous block, you will have problems long before that under ASP.NET, memory fragmentation can make even 10 MB objects problematic. – Gary Walker Oct 30 '13 at 20:43
  • Alexi, I was assuming Cassini based on original description, maybe a bad assumption – Gary Walker Oct 30 '13 at 20:46
  • At what point do you get that exception? Edit your question and post the *exact* exception along with the full stack trace, and show us what line of code gives that exception. – Jim Mischel Oct 30 '13 at 21:03
  • Does it not error out if you don't have the debugger attached? The CLR will keep objects in memory a lot longer than it needs to when a debugger is attached (any object you could potentially list in the `watch` window needs to be kept in memory and not allowed to be garbage collected) – Scott Chamberlain Oct 30 '13 at 21:39
  • Will this run fine when deployed under IIS7.5 or similar? Can I config the app pool some how to bump the memory allocation limit? – Bill Oct 30 '13 at 22:01

2 Answers2

7

This DID FIX the problem. I'm still limited to 2 GB of memory from within VS 2013 in debug mode. However, when deploying to IIS7.5, 2008 R2 Server, and App. Pool of dot net 4.0x, I can use a lot more memory for my website. It took the following setting in my web.config. I now need to bump my physical from 8 to 16 GBs; But, that is a different story. Solution:

<gcAllowVeryLargeObjects enabled="true" />

https://msdn.microsoft.com/en-us/library/hh285054(v=vs.110).aspx

Burak Karakuş
  • 1,368
  • 5
  • 20
  • 43
Bill
  • 915
  • 2
  • 13
  • 23
0

Between BeginUpdate() and EndUpdate(), Garbage collection fixed the problem for me in 32 and 64 bit. GC.Collect(); GC.WaitForPendingFinalizers();

Soumen
  • 1