VTDGen throws ParseException: File too big

Question

I am trying to parse xml file with VTDGen library. It was perfect to parse xml till I am having trouble with over 1GB xml File.

This is a code how I parse it.

            VTDGen vg = new VTDGen();
            in = new SmbFileInputStream(fileToGet);
            byte[] b = new byte[(int) fileToGet.length()];
            in.read(b);             
            vg.setDoc(b);
            vg.parse(true);

This is an error I get it.

com.ximpleware.ParseException: Other error: file size too big >=1GB

Is there any way I can increase size or should I write a code with a another parser?

Thank you in advance.

I thïk you should disable namespace awareness. Ie parse(false) or use extended vtd-xml — vtd-xml-author, Dec 21 '13 at 05:29

score 1 · Answer 1 · answered Dec 20 '13 at 18:58

1

read about the limitations of VTD:

Upper limits of various fields: (1) For starting tags (the max Qname length is 2048; the prefix 512), overflow conditions result in parse exceptions. For other tokens (upper limit is 1M), one can potentially break a long token into multiple shorter ones.(2) Depth field overflow condition results in parse exceptions. (3) Starting offset: Currently the biggest document supported is 1G characters (1G bytes or 2G bytes, depending on actual document encoding).

From http://vtd-xml.sourceforge.net/userGuide/0.html

answered Dec 20 '13 at 18:58

Mathis Hobden

356
2
7
19

I guess I have to write a new parser any recommendation for parser that handles over 1GB? – draford Dec 20 '13 at 19:31
you need to use a SAX based parser, because SAX parse data sequentially not like DOM based parsers which put the entire content in the virtual memory see this http://stackoverflow.com/questions/3969713/java-xml-parser-for-huge-files – Mathis Hobden Dec 20 '13 at 23:56

score 0 · Answer 2 · answered Feb 17 '16 at 02:20

There are two ways to get around the issue:

Use extended VTD-XML. It is part of the vtd-xml distribution, shares a very similar API, but is a standalone product by itself.
Turn off namespace awareness, that will boost the max document size from 1 GB to 2GB

VTDGen throws ParseException: File too big

2 Answers2