Performance issues creating an XML file from Database

Question

I'm trying to write an XML file, which is a Google Product feed, but it's very slow.

Having done quite a bit of Googling, I am not sure of the fastest approach, except for breaking it into smaller files.

I have about 3500 products currently, and this is what I am doing.

Dim fs As New FileStream(HttpContext.Current.Server.MapPath("LOCATION OF FEED XML"), FileMode.Create)
Dim writer As New XmlTextWriter(fs, System.Text.Encoding.UTF8)

writer.WriteStartDocument()
   ' ==== LOAD DATABASE INTO OdbcDataReader CALLED 'd' ==== '
While d.Read
   try
        writer.WriteStartElement("item")

        writer.WriteStartElement("g:id")
        writer.WriteString(pID)
        writer.WriteEndElement()
        '==== etc 
    catch
        ignore broken ones
    end
End While

     writer.WriteEndElement()
     writer.WriteEndDocument()
     writer.Close()

 d.Dispose()
 connection.Dispose()
 connection.Close()

I am not familiar with FileStream, MemoryStream etc, so not really sure what is exactly happening here where the writing is concerned.

Presumably writing to memory would be faster, but what's the process to do that and then save to disk?

Would the amount of data be an issue for memory?

Obviously I am coding with VB, using .NET 2.0

Any suggestions appreciated.

score 0 · Accepted Answer · answered Jul 09 '12 at 14:13

0

The performance problem doesn't seem to be in writing, but looks like it is the reading that is causing the slow performance. Where are you reading the data from? Database? Then that's where the performance bottleneck is.

answered Jul 09 '12 at 14:13

oazabir

1,599
9
15

Thanks. So are you saying that the write method I am using is the best method for performance? Is there a faster method to iterate a dataset than I am using then? I'm using a MySQL Database and `OdbcDataReader` my query's loading faster enough, so it's not loading the data that's slow, but something in the process of writing it out. – Jamie Hartnoll Jul 09 '12 at 14:39
Oh, I'm such an idiot! I've just realised in getting my shipping data I was looping back to the database whilst reading each record! Doh! It's still taking about 60 seconds to process, but that's a 50% improvement and I think I've spotted a couple of extra issues too! Thanks! – Jamie Hartnoll Jul 09 '12 at 15:35

Performance issues creating an XML file from Database

1 Answers1