How to get 1000000 records and write xml feed to a file?

Question

I would like to create a c# console application to get about 1000000 rows of data.

add some filtering logic in code and generate xml feed.

The one I'm using is working fine but for 250K rows of the data, than I get out of memory exception.

Here is an example of code I use in web application I need to change it console application and make it efficient.

var xrFeed = new XmlTextWriter(File.Create(@"c:\Items.xml"), Encoding.UTF8);

xrFeed.WriteStartDocument();
xrFeed.WriteStartElement("Name");

IEnumerable<ItemClass> items = _source.GetItems();

if (items != null)
{ 
    foreach (var i in items)
    {                    
        xrFeed.WriteStartElement("ad");

            xrFeed.WriteStartElement("id");
            xrFeed.WriteCData(m.ListingId.ToString());
            xrFeed.WriteEndElement();

            xrFeed.WriteStartElement("firstParameter");
            xrFeed.WriteCData("parameter");
            xrFeed.WriteEndElement();

            xrFeed.WriteStartElement("secondParameter");
            xrFeed.WriteCData("parameter2");
            xrFeed.WriteEndElement();

            xrFeed.WriteStartElement("thirdParameter");
            xrFeed.WriteCData("parameter3");
            xrFeed.WriteEndElement();

        xrFeed.WriteEndElement();
    }

    xrFeed.WriteEndElement();
    xrFeed.WriteEndDocument();

    xrFeed.Flush();
    xrFeed.Close();

    Response.End();
    DataBind();         
}

What version of .NET are you using? You should not be using `new XmlTextWriter()` at all (since .NET 2.0), and you might do better using LINQ to XML using [XStreamingElement](http://msdn.microsoft.com/en-us/library/system.xml.linq.xstreamingelement.aspx). — John Saunders, May 27 '11 at 14:28
Where do you get the OOM exception? While looping through data rows, or while reading the data? What is `_source`? — CodingWithSpike, May 27 '11 at 14:34
I wouldn't be surprised if the `DataBind()` throws the exception. — H H, May 27 '11 at 15:23
I'm using .net 3.5 - 4.0. Could you please show me an example of more efficient way. I'm getting the exception while looping — Michael Born, May 27 '11 at 15:26
@Michael: and if you comment-out the DataBind? If possible post a stacktrace (any text from the Exception). — H H, May 27 '11 at 15:30
You need to flush the file more often. You are trying to place the entire file in memory which is a horrible idea even if you had enough memory to do so. — Security Hound, May 27 '11 at 15:30

Magnus · Accepted Answer · 2011-05-27T14:39:51.340

2

Try flushing the writer every 1000 items or so. Also you might want to partially retrieve the data from your datasource.

edited May 27 '11 at 14:39

answered May 27 '11 at 14:27

Magnus

45,362
8
80
118

1

Just try? Use a counter, check if its 1000, then flush the file and reset counter. – Security Hound May 27 '11 at 15:30
Do as @Ramhound suggests. To partially retrieve the data from your datasource, it depends on what kind of datasource it is. Linq? Than use .Skip(X).Take(Y) – Magnus May 28 '11 at 11:02

How to get 1000000 records and write xml feed to a file?

1 Answers1