Large XML reading using XMLTextReader is very slow

Question

I like to think that my problem is not very unique, given size of my XML file is just 3MB. There are close to 60 thousand records in XML. I am having hard time reducing the processing the processing time. Currently it is taking close to 7-8 minutes to read and insert into Datatables. (Please note I am NOT inserting in database yet, so database transactions are not the issues here)

Here is the code I wrote. Any suggestion to reduce the processing time will be greatly appreciated.

 XmlTextReader reader = new XmlTextReader(destFile);            

        while (reader.Read())
        {
            switch (reader.NodeType)
            {
                case XmlNodeType.Element:
                    if (reader.Name == "Report")
                    {                           
                        FileDataTable = UpdateReportTable(FileDataTable, reader);

                    else if (reader.Name == "Name")
                    {
                        NameTable = UpdateNameTable(NameTable, reader);
                    }

                    else if (reader.Name == "Entries")
                    {
                        EntriesTable = UpdateEntriesTable(EntriesTable , reader);
                    }                     

                    reader.MoveToElement();
                    break;
                case XmlNodeType.Text:
                    break;
                case XmlNodeType.EndElement:
                    break;
            }
        }

Then I have following function to get value into DataTable. Element "Entries" are taking 90% of the time so I am posting that code, other functions are similar.

private static DataTable UpdateEntriesTable(DataTable entries, XmlTextReader reader)
    {
        DataRow row = entries.NewRow();

        for (int attInd = 0; attInd < reader.AttributeCount; attInd++)
        {
            reader.MoveToAttribute(attInd);
            if (reader.Name == "refDataId") { row["DataId"] = Convert.ToInt32(reader.Value); }

        }

        reader.MoveToElement();
        reader.Read();
        row["DataCount"] = Convert.ToInt32(reader.Value);
        row["LastModifiedOn"] = DateTime.Now;
        try
        {
            entries.Rows.Add(row);
            entries.AcceptChanges();
        }
        catch (Exception ex)
        {
            log.Error(ex.Message);
            return entries;
        }
        return entries;
    }

Instead of doing `entries.AcceptChanges();` row by row, you should do a bulk operation. So add all your rows to your `DataTable` and do `entries.AcceptChanges();` only once at the end. — Jürgen Müller, Oct 22 '18 at 15:20
U left is as a "comment" so I can't accept it as an answer. Thanks. — user2918107, Oct 22 '18 at 15:40

score 1 · Answer 1 · answered Oct 22 '18 at 15:24

It looks like you're saving each entity to the database as you go. That can be pretty slow going, especially if you have to open a connection, save data, and then close the connection again.

I'd suggest attempting to wrap all the entity changes up in a bulk update, so you're only having to open a connection to the database and write to it once. You can add all of your entities to the DataTable and after you're done processing then execute AcceptChanges();. That would likely save you a ton of time.

Large XML reading using XMLTextReader is very slow

1 Answers1