How to merge >1000 xml files into one using Java

Question

I am trying to merge many xml files into one. I have successfully done that in DOM, but this solution is limited to a few files. When I run it on multiple files >1000 I am getting a java.lang.OutOfMemoryError.

What I want to achieve is where i have the following files

file 1:

<root>
....
</root>

file 2:

<root>
......
</root>

file n:

<root>
....
</root>

resulting in: output:

<rootSet>
<root>
....
</root>
<root>
....
</root>
<root>
....
</root>
</rootSet>

This is my current implementation:

    DocumentBuilderFactory docFactory = DocumentBuilderFactory.newInstance();
    DocumentBuilder docBuilder = docFactory.newDocumentBuilder();
    Document doc = docBuilder.newDocument();
    Element rootSetElement = doc.createElement("rootSet");
    Node rootSetNode = doc.appendChild(rootSetElement);
    Element creationElement = doc.createElement("creationDate");
    rootSetNode.appendChild(creationElement);
    creationElement.setTextContent(dateString); 
    File dir = new File("/tmp/rootFiles");
    String[] files = dir.list();
    if (files == null) {
        System.out.println("No roots to merge!");
    } else {
        Document rootDocument;
            for (int i=0; i<files.length; i++) {
                       File filename = new File(dir+"/"+files[i]);        
               rootDocument = docBuilder.parse(filename);
               Node tempDoc = doc.importNode((Node) Document.getElementsByTagName("root").item(0), true);
               rootSetNode.appendChild(tempDoc);
        }
    }

I have experimented a lot with xslt, sax, but I seem to keep missing something. Any help would be highly appreciated

Is there any reason you need to actually keep the DOM in memory? Do you need more than a simple string concatenation in this case? — Yuval Adam, May 25 '12 at 18:42
simple concatenation would keep the xml declaration if each individual xml file being merged. Bit indeed in principle I am looking for a simple concatenation of xml files. — Andra, May 25 '12 at 18:44
Why not put multiple XML files into one archive? It ends up as one file. Make it uncompressed if read/write speed is important, compressed if file size or bandwidth is more important. — Andrew Thompson, May 25 '12 at 18:45

score 10 · Accepted Answer · answered May 25 '12 at 19:22

You might also consider using StAX. Here's code that would do what you want:

import java.io.File;
import java.io.FileWriter;
import java.io.Writer;

import javax.xml.stream.XMLEventFactory;
import javax.xml.stream.XMLEventReader;
import javax.xml.stream.XMLEventWriter;
import javax.xml.stream.XMLInputFactory;
import javax.xml.stream.XMLOutputFactory;
import javax.xml.stream.events.XMLEvent;
import javax.xml.transform.stream.StreamSource;

public class XMLConcat {
    public static void main(String[] args) throws Throwable {
        File dir = new File("/tmp/rootFiles");
        File[] rootFiles = dir.listFiles();

        Writer outputWriter = new FileWriter("/tmp/mergedFile.xml");
        XMLOutputFactory xmlOutFactory = XMLOutputFactory.newFactory();
        XMLEventWriter xmlEventWriter = xmlOutFactory.createXMLEventWriter(outputWriter);
        XMLEventFactory xmlEventFactory = XMLEventFactory.newFactory();

        xmlEventWriter.add(xmlEventFactory.createStartDocument());
        xmlEventWriter.add(xmlEventFactory.createStartElement("", null, "rootSet"));

        XMLInputFactory xmlInFactory = XMLInputFactory.newFactory();
        for (File rootFile : rootFiles) {
            XMLEventReader xmlEventReader = xmlInFactory.createXMLEventReader(new StreamSource(rootFile));
            XMLEvent event = xmlEventReader.nextEvent();
            // Skip ahead in the input to the opening document element
            while (event.getEventType() != XMLEvent.START_ELEMENT) {
                event = xmlEventReader.nextEvent();
            }

            do {
                xmlEventWriter.add(event);
                event = xmlEventReader.nextEvent();
            } while (event.getEventType() != XMLEvent.END_DOCUMENT);
            xmlEventReader.close();
        }

        xmlEventWriter.add(xmlEventFactory.createEndElement("", null, "rootSet"));
        xmlEventWriter.add(xmlEventFactory.createEndDocument());

        xmlEventWriter.close();
        outputWriter.close();
    }
}

One minor caveat is that this API seems to mess with empty tags, changing <foo/> into <foo></foo>.

Mattias Isegran Bergander · Answer 2 · 2012-05-25T20:43:28.300

Just do it without any xml-parsing as it doesn't seem to require any actual parsing of the xml.

For efficiency do something like this:

File dir = new File("/tmp/rootFiles");
String[] files = dir.list();
if (files == null) {
    System.out.println("No roots to merge!");
} else {
        try (FileChannel output = new FileOutputStream("output").getChannel()) {
            ByteBuffer buff = ByteBuffer.allocate(32);
            buff.put("<rootSet>\n".getBytes()); // specify encoding too
            buff.flip();
            output.write(buff);
            buff.clear();
            for (String file : files) {
                try (FileChannel in = new FileInputStream(new File(dir, file).getChannel()) {
                    in.transferTo(0, 1 << 24, output);
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
            buff.put("</rootSet>\n".getBytes()); // specify encoding too
            buff.flip();
            output.write(buff);
        } catch (IOException e) {
            e.printStackTrace();
        }

score 2 · Answer 3 · answered May 25 '12 at 18:50

2

DOM needs to keep the whole document in memory. If you don't need to do any special operation with your tags, I would simply use an InputStream and read all the files. If you need to do some operations, then use SAX.

answered May 25 '12 at 18:50

Carlos Tasada

4,438
1
23
26

score 2 · Answer 4 · answered May 25 '12 at 19:00

Dom does consume a lot of memory. You have, imho, the following alternatives.

The best one is to use SAX. Using sax, only a very small amount of memory is used, cause basically nearly a single element is travelling from input to output at any given time, so memory footprint is extremely low. However, using sax is not so simple, cause compared to dom it is a bit counterintuitive.

Try Stax, not tried myself, but it's a kind of sax on steroids easier to implement and use, cause as opposed to just receiving sax events you don't control, you actually "ask the source" to stream you the elements you want, so it fits in the middle between dom and sax, has a memory footprint similar to sax, but a more friendly paradigm.

Sax, stax, dom are all important if you want to correctly preserve, declare etc... namespaces and other XML oddities.

However, if you just need a quick and dirty way, which will probably be namespace compliant as well, use plain old strings and writers.

Start outputting to the FileWriter the declaration and the root element of your "big" document. Then load, using dom if you like, each single file. Select the elements you want to end up in the "big" file, serialize them back to a string, and send the to the writer. the writer will flush to disk without using enormous amount of memory, and dom will load only one document per iteration. Unless you also have very big files on the input side, or plan to run it on a cellphone, you should not have a lot of memory problems. If dom serializes it correctly, it should preserve namespace declarations and the like, and the code will be just a bunch of lines more than the one you posted.

alain.janinm · Answer 5 · 2012-05-25T19:50:58.057

For this kind of work I will suggest not to use DOM, reading the file content and making a substring is simpler and enough.

I'm thinking of something like that :

String rootContent = document.substring(document.indexOf("<root>"), document.lastIndexOf("</root>")+7);

Then to avoid to much memory consummation. Write in the main file after every xml extraction with a BufferedWritter for example. For better performance you can also use java.nio.

Malcolm Smith · Answer 6 · 2012-05-26T11:40:38.187

I think what you're doing is valid. The only way to make it scale to really huge numbers of files is to use a text based approach with streaming, so you never keep the whole thing in memory. But, hey! Good news. Memory is cheap these days, and 64bit JVMs are all the rage, so maybe all you need is to increase the the heap size. Try re-running your program with a -Xms1g JVM option (allocates 1Gb initial heap size).

I also tend to use XOM for all my DOM requirements. Give it a go. Much more efficient. Don't know for sure on the memory requirements, but its orders of magnitude faster in my experience.

How to merge >1000 xml files into one using Java

6 Answers6

Linked