1

I need to evaluate multiple XPath (or possibly XQuery - I have some freedom to change the design here) expressions over a larger number of huge XML documents, potentially gigabyte size. If the files were small I could easily evaluate the expressions one by one with a DOM tree. If there was only one expression I might be able to evaluate it in streaming mode. However, I have found no solution for efficiently evaluating multiple expressions in streaming mode, i.e. without making multiple passes.

There are research papers. XTREAM looks pretty good, but though the paper was written in 2005 I can find no implementation. This is even older, but still I can find no implementation.

Is there a library (ideally in Java and ideally open source) that can do this?

ewramner
  • 5,810
  • 2
  • 17
  • 33
  • 1
    BaseX and eXist-db are open-source XML databases that support XQuery 3.1 so you could check how they perform/evaluate the queries you have in mind. In terms of streaming, Saxon EE (not open source) supports streaming XSLT 3.0 (a W3C standard since 2017) and also allows some extensions to XQuery to do some XQuery with streaming. – Martin Honnen Nov 26 '22 at 12:03
  • 1
    For Saxon the relevant documentation is https://www.saxonica.com/html/documentation11/streaming/xslt-streaming.html for XSLT and https://www.saxonica.com/html/documentation11/streaming/streamed-query.html for XQuery. – Martin Honnen Nov 26 '22 at 12:05

1 Answers1

1

With XSLT 3.0 streaming (which in practice means Saxon-EE [my company's product], since EXSELT seems to have gone off-air), you can use xsl:fork to evaluate multiple streaming XPath expressions in a single pass over the input, for example

<xsl:source-document href="input.xml">
  <xsl:fork>
    <xsl:sequence>
      <xsl:result-document href="out1.xml">
       <out1>{count(//a)}</out1>
      </xsl:result-document>
    </xsl:sequence>
    <xsl:sequence>
      <xsl:result-document href="out2.xml">
       <out2>{count(//b)}</out1>
      </xsl:result-document>
    </xsl:sequence>
  </xsl:fork>
</xsl:source-document>

To run this over multiple source documents, you can use <xsl:for-each select="collection(....)"/> and with Saxon-EE you can add saxon:threads="n" to process multiple inputs in parallel.

Sorry, this isn't open source - this isn't the kind of technology that you can implement in a spare weekend.

Michael Kay
  • 156,231
  • 11
  • 92
  • 164
  • Cool. We would need the redistribution license, which puts it way out of my price range, but it is nice to know that the option exists. – ewramner Nov 26 '22 at 17:25