Fastest way to parse flat, attribute-heavy xml in Java or Scala

Question

If I have a big xml file like the following. What would be the fastest way to parse it in Java or Scala. Streaming individual elements is important but not absolutely essential

All I'm interesting in is getting the attribute values from each result object.

<Response>
    <Result att1="1", att2="2", att3="3", att4="4", att5="5"/>
    <Result att1="1", att2="2", att3="3", att4="4", att5="5"/>
    <Result att1="1", att2="2", att3="3", att4="4", att5="5"/>
    <Result att1="1", att2="2", att3="3", att4="4", att5="5"/>
</Response>

I want to parse the value of all attributes in the result elements — user2668128, Aug 12 '13 at 14:09
The question is ambiguous and could be framed better, but closing it as "primarily opinion-based" doesn't make much sense to me. — Travis Brown, Aug 12 '13 at 21:11

cmbaxter · Answer 1 · 2013-08-12T21:23:15.180

7

If your file is large and you don't want to load the whole thing into memory (i.e. DOM), then one path you could take is the pull parsing route. If you want to do pull parsing in scala, looking for the "start element" event in order to inspect the attributes, then you could do something like this:

import scala.io.Source
import java.io.File
import scala.xml.pull.XMLEventReader
import scala.xml.pull.EvElemStart

val src = Source.fromFile(new File(pathToXml))
val reader = new XMLEventReader(src)
reader foreach{ 
  case EvElemStart(_, _, attrs, _) =>
    //do something here

  case _ =>
}

Following this approach should ensure that your file is not read into memory and should be fast.

edited Aug 12 '13 at 21:23

answered Aug 12 '13 at 16:29

cmbaxter

35,283
4
86
95

1

This isn't SAX parsing, it's pull parsing (you should mention the package, by the way). It's more similar to Java's StAX API than to SAX, but it's not (directly) built on either. – Travis Brown Aug 12 '13 at 21:00
@TravisBrown, thanks for the heads up. Answer has been edited. – cmbaxter Aug 12 '13 at 21:23

score 2 · Accepted Answer · edited May 23 '17 at 11:44

Scala XML (can be slow & memory hungry)

The answer from cmbaxter is technical correct, but it can be improved with the "flatMap that shit" pattern :-)

    import io.Source
    import xml.pull._

    // Make it "def", because the Source is stateful and may be exhausted after it is read
    def xmlsrc=Source.fromString("""<Response>
         |     <Result att1="1" att2="2" att3="3" att4="4" att5="5"/>
         |     <Result att1="1" att2="2" att3="3" att4="4" att5="5"/>
         |     <Result att1="1" att2="2" att3="3" att4="4" att5="5"/>
         |     <Result att1="1" att2="2" att3="3" att4="4" att5="5"/>
         | </Response>""")

    // Also as "def", because the result is an iterator that may be exhausted
    def xmlEr=new XMLEventReader(xmlsrc)

    // flatMap keeps the "outer shape" of the type it operates, so we are still dealing with an iterator

    def attrs = xmlEr.flatMap{
         |   case e : EvElemStart => e.attrs.map(a => (a.key, a.value))
         |   case _ => Iterable.empty
         | }

    // Now lets look what is inside:
    attrs.foreach(println _)

    // Or just let's collect all values from "att5"
    attrs.collect{ case (name, value) if name == "att5" =>value}.foreach(println _)

Scales XML (faster & needs less memory)

But this will not be fastest way. The Scala API is quite slow and memory hungry compared to other solutions, like benchmarks show. But fortunately there's a faster and less memory hungry solution:

    import scales.utils._
    import ScalesUtils._
    import scales.xml._
    import ScalesXml._
    import java.io.StringReader

    def xmlsrc=new StringReader("""<Response>
         |     <Result att1="1" att2="2" att3="3" att4="4" att5="5"/>
         |     <Result att1="1" att2="2" att3="3" att4="4" att5="5"/>
         |     <Result att1="1" att2="2" att3="3" att4="4" att5="5"/>
         |     <Result att1="1" att2="2" att3="3" att4="4" att5="5"/>
         | </Response>""")
    def pull=pullXml(xmlsrc)
    def attributes = pull flatMap {
         |   case Left(elem : Elem) => elem.attributes
         |   case _ => Nil
         | } map (attr => (attr.name, attr.value))

    attributes.foreach(println _)

Don't forget to close you iterators after you are done with them. Here it is not necessary, because I am working with a StringReader.

Anti XML

There is also the Anti XML library, which looks quite nice in benchmarks and seems to have a very nice API. Unfortunately I could not get it to run with Scala 2.10, so I cannot provide a running example.

Conclusion

With the examples above, you should be able to write a small test application. With these you can run your own benchmarks. Looking on the benchmarks quoted above, I guess that Scales XML might solve your problem. But without real meassuring, this is really only a guess.

Benchmark yourself and perhaps you can post your results.

Fastest way to parse flat, attribute-heavy xml in Java or Scala

2 Answers2

Scala XML (can be slow & memory hungry)

Scales XML (faster & needs less memory)

Anti XML

Conclusion

Linked