OutOfMemoryException at StringWriter.ToString()

Question

I am getting an OutOfMemoryException when calling ToString on my StringWriter:

StringWriter stringWriter = new System.IO.StringWriter();
XmlSerializer serializer = new XmlSerializer(typeof(T));
serializer.Serialize(stringWriter, data);
string xmlString = stringWriter.ToString(); // <-- Exception occurs here

How can I solve this issue?

Answer 1

Try this code. It uses a file as a temporary buffer.

List<Dummy> lst = new List<Dummy>();

        for (var i = 0; i < 100000; i++)

        {
            lst.Add(new Dummy()
                    {
                        X =  i,
                        Y =  i * 2
                    });

        }

        XmlSerializer serializer = new XmlSerializer(typeof(List<Dummy>));

        // estimate your memory consumption ... it would be around 4 bytes reference + 4 bytes object type pointer + 8 bytes those ints + let's say another 4 bytes other hidden CLR metadatas. a total of 20 bytes per instance + 4 bytes reference to our object (in the list array) => around 24 bytes per instance. Round up to a let's say 50 bytes per instance. Multiply it by 100.000 = 5.000.000

        MemoryStream memStream = new MemoryStream(5000000);

        serializer.Serialize(memStream, lst);
        memStream.Position = 0;

        string tempDatafileName = null;
        var dataWasWritten = false;
        try
        {
            var fileName = Guid.NewGuid().ToString() + ".tempd";
            var specialFolderPath = Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData);

            using (var fs = new FileStream(tempDatafileName, FileMode.Create, FileAccess.ReadWrite))
                memStream.WriteTo(fs);

            dataWasWritten = true;

            memStream.Dispose();
            memStream = null;

            lst.Clear();
            lst = null;
            // force a full second generational GC
            GC.Collect(2);

            // reading the content in string
            string myXml = File.ReadAllText(tempDatafileName);
        }
        finally
        {
            if (dataWasWritten && string.IsNullOrWhiteSpace(tempDatafileName) == false)
            {
                if (File.Exists(tempDatafileName))
                {
                    try
                    {
                        File.Delete(tempDatafileName);
                    }
                    catch
                    {

                    }
                }
            }
        }

Answer 2

I am sure that stringWriter is already very large, let's say a size more than 1 Gb and that your computer has a total memory of 2 GB. When you call ToString () the memory will get doubled because a new string will be created and allocated in heap of size 1GB to be able to copy the content of stringWriter. Try to avoid the unnecesary ToString () and try to do what you need to do using directly the stringWriter. Avoiding ToString(), normally you would decrease memory footprint by 50%.

But if you really need your data as string and you don't have sufficient memory, try to save the content first in a file, dispose the StringWriter and load the file content into a string using a StreamReader.ReadToEnd () API.

Another aproach would be to try to cut un chunks your serialized data and try to parse it in chunks. For example, let's consider an XML that has a structure like this:

<Root>
    <Item>some data 1</Item>
    <Item>some data 2</Item>
    <Item>some data 3</Item>
    ....
    <Item>some data n</Item>
</Root>

You could serialize your object into a MemoryStream, then read it in chunks and create "little" XML data from your chunks that looks like this:

First xml:

<Root>
    <Item> some data 1 </Item>
</Root>

Second xml:

 <Root>
    <Item> some data 2 </Item>
</Root>

And so on, that could be checked individually and validated.

Answer 3

this should do the trick :

using (StreamWriter sw = File.AppendText(path))
{
    using (StreamReader sr = new StreamReader(ms))
    {
        while (sr.Peek() >= 0)
        {
            string l = sr.ReadLine();

            sw.WriteLine(l);
        }
    }
}

Answer 4

It is working when doing like this..

   using (MemoryStream ms = new MemoryStream())
            {
                serializer.Serialize(ms, data);
                ms.Position = 0;
                XmlReader reader = XmlReader.Create(ms, schemaReaderSettings);
            }