0

I have a process that reads an XML file. It uses the XMLTextReader class to do this as it is supposed to be a fast, forward only XML parser/reader.

Works just great with a 1 megabyte test file but comes to a complete halt when working on a 12 meg file in the live system.

Are there any solutions to this other than writing my own XML reader? That's not the end of the world but I would prefer to use available standard components if possible

Joel Coehoorn
  • 399,467
  • 113
  • 570
  • 794
user6690
  • 31
  • 1
  • 2

6 Answers6

1

SAXExpat used to be really good. Expat is the XML parser, almost a reference implementation. I remember using it to read some synchronization XML files sent over a TCP connection, sometimes really big files (around 50mb) without any kind of problem. And that was 3/4 years ago, in .NET 1.1 and really crappy computers.

dguaraglia
  • 5,774
  • 1
  • 26
  • 23
1

I would be very surprised if the problem were in the XmlTextReader.

If you spend a few minutes to write a test program that creates an XmlTextReader and simply uses Read() to read through each node in the file until it gets to the end of the document, I bet you'll find that it zooms through your 12mb file like a hot knife through butter. That's the first thing I'd try if I were experiencing this problem.

Because once you've eliminated XmlTextReader as the source of the problem, you can focus your attention on what's actually causing it - which is, very probably, the code that processes the nodes that you're reading, not the code that reads the nodes.

Robert Rossney
  • 94,622
  • 24
  • 146
  • 218
0

I have had similar performance issues in the past. I traced it back to trying to remotely resolve against a DTD/schema. Are you doing this? Try setting XmlTextReader.XmlResolver to null if possible.

Ash
  • 60,973
  • 31
  • 151
  • 169
0

Depends what you do with what you get out of the reader. Are you putting it in an XML DOM, or any object model for that matter? That would make a big memory hit not matter what language or library you use.

Maybe it is flawed in 1.1, thought about trying out 2.0? I never used the XmlTextReader in my 1.1 days, so I can't vouch for it: but since 2.0 it is perfect.

Jonathan C Dickinson
  • 7,181
  • 4
  • 35
  • 46
  • Hi - the data is being posted to a SQLServer 2000 database. The process is running on a four processor server with plenty of memory. I am not in a position to change the Framework version I am using on this project as it is part of an existing application written a while back. – user6690 Oct 02 '08 at 13:28
  • Man, legacy sucks huh? Let me get this right, do you for instance say if (reader.LocalName == "insert-user") InsertIntoDB(reader.ReadInnerXml()) (in more lines)? If you put this in a DOM at *all* you are going to suffer. – Jonathan C Dickinson Oct 08 '08 at 13:17
0

Just one thought. Are you opening a database transaction for the length of the entire process? If so try it without the transaction or at least commit more often during the process.

Darrel Miller
  • 139,164
  • 32
  • 194
  • 243
0

I hate to recommend this, but if the software isn't sold or external, you could try bringing in the reader from Mono and see if that fixes your woes.

Jonathan C Dickinson
  • 7,181
  • 4
  • 35
  • 46