0

I have a very large file that is not on the same box as the Marklogic server. Putting the file on the same server is not an option.

What is the best way to load the file into the database? I'm thinking that a SAX parser could pick off nodes and load them into the database.

<a>
  <b>xxx</b>
  <b>yyy</b>
  <b>zzz</b>
</a>

So, using the above xml I'd create a document in Marklogic containing . Then, using the SAX parser, I'd grab the first <b> element and insert it as child of <a>. I'd repeat for all remaining <b> elements.

Does that sound like the best approach? Would it be performant?

TIA

Sofia
  • 771
  • 1
  • 8
  • 22
Todd
  • 2,829
  • 5
  • 34
  • 56

3 Answers3

2

As Eric indicates, one large document is probably not what you want. MarkLogic is designed to work best with many (thousands to billions) of bite-sized (dozens to hundreds of kilobytes) documents. If you’re familiar with relational databases, you can think of relational rows (not tables) as roughly equivalent to documents in a MarkLogic database.

Can you provide some more details on what you’re trying to do? What types of queries do you expect to perform? What does your data actually look like? How large is “very large”?

Justin Makeig
  • 2,097
  • 15
  • 29
  • The size can vary, so there isn't any good "one answer", but a sample document I'm working on in testing is half a gig in size. When trying to load as a string, I get a heap space error. The document is only intended for a very short life - I query to dedupe it, then return select values from each of the repeating elements. I'm beginning to think just getting those values from SAX rather than dealing with Marklogic is the correct answer. – Todd Nov 30 '12 at 13:28
1

First, once you have all that data in MarkLogic, you're not likely going to want it all in one Document.

You can use a tool like mlcp to help you break up your doc and load it. See http://developer.marklogic.com/products/mlcp as well as http://docs.marklogic.com/guide/ingestion and in particular the bit on mlcp (content pump) at http://docs.marklogic.com/guide/ingestion/content-pump

Eric Bloch
  • 2,882
  • 2
  • 20
  • 26
  • Eric, the document will be very short lived, so its size inside the database isn't a major concern. In answering Justin, I think I may have found a better solution (using SAX for what I'm after rather than Xquerying Marklogic). Thanks for the suggestion though. I'll check that out for other uses. – Todd Nov 30 '12 at 13:34
  • Even if the document won't live for long, the size in the database is an issue as large documents can blow out the size of your tree cache. – Eric Bloch Dec 02 '12 at 20:48
0

You may find the StAX scripting features of xmlsh useful to you. This parses like SAX but is much easier to deal with because its a PULL not a PUSH technology so no callbacks. And its fully scriptable so much less code then pure Java, while being nearly as fast.

http://www.xmlsh.org/FunctionsStAX

-David

DALDEI
  • 3,722
  • 13
  • 9