Splitting Large XML FIles Java (StAX)

Question

I am using this code https://github.com/koen-serneels/blog/tree/master/BigXmlTest/src/main/java/be/error/bigxmltest from GitHub to split large XML files(6GB).I am starting with the basic XML File provided to understand the program. The program outputs the header for each file that is split. I am Trying to add a Footer Element now also for each file and having some difficulties, any suggestions or advice on how to go about this would be great.

I have added The footer to the schema and written the code to add a footer in the "openOutputFileAndWriteHeader()" method. However This adds the footer within the Content Element. Shown Below:

private XMLEventWriter openOutputFileAndWriteHeader(int fileNumber) throws Exception
{

    XMLEventFactory xmlEventFactory = XMLEventFactory.newInstance();
    XMLOutputFactory xmlOutputFactory = XMLOutputFactory.newInstance();
    xmlOutputFactory.setProperty(XMLOutputFactory.IS_REPAIRING_NAMESPACES, true);
    XMLEventWriter writer = xmlOutputFactory.createXMLEventWriter(new FileOutputStream(new File("C:\\Users\\thamm\\Desktop\\SplitFiles\\File_" + fileNumber + ".xml")));

    writer.setDefaultNamespace(DOCUMENT_NS);
    writer.add(xmlEventFactory.createStartDocument());

    writer.add(xmlEventFactory.createStartElement(new QName(DOCUMENT_NS, ROOT_ELEMENT), null, null));

    HeaderType header = objectFactory.createHeaderType();
    header.setSomeHeaderElement("List of Employees");
    marshaller.marshal(new JAXBElement<HeaderType>(new QName(DOCUMENT_NS, HEADER_ELEMENT, ""), HeaderType.class, 
                    HeaderType.class, header), writer);

    writer.add(new StartElementEvent(new QName(DOCUMENT_NS, CONTENT_ELEMENT)));

    FooterType footer = objectFactory.createFooterType();
    footer.setSomeFooterElement("Footer");
    marshaller.marshal(new JAXBElement<FooterType>(new QName(DOCUMENT_NS, FOOTER_ELEMENT,""), FooterType.class, 
                      FooterType.class, footer), writer);

    return writer;
}

Output:

<?xml version='1.0' encoding='UTF-8'?>
<BigXmlTest xmlns="http://www.sidero.ie/bigxmltest">
    <Header>
        <SomeHeaderElement>List of Employees</SomeHeaderElement>
    </Header>
    <Content>
        <Footer>
            <SomeFooterElement>Footer</SomeFooterElement>
        </Footer>
        <Data>Data4</Data>
        <Data>Data5</Data>
        <Data>Data6</Data>
    </Content>
</BigXmlTest>

6GB is that a single file? – vtd-xml-author Mar 31 '17 at 20:43 — vtd-xml-author, Mar 31 '17 at 20:43

score 0 · Accepted Answer · answered Mar 30 '17 at 11:24

Name of method openOutputFileAndWriteHeader hints that it will create new file. It's not proper place for footer.

In BigXmlTestIteratorApi.java from line 74 you can see code:

xmlEventWriter.close(); // Also closes any open Element(s) and the document
xmlEventWriter = openOutputFileAndWriteHeader(++fileNumber); // Continue with next file
dataRepetitions = 0;

To add footer, you need add something before closing file:

writeFooter(footer);
xmlEventWriter.close(); 
xmlEventWriter = openOutputFileAndWriteHeader(++fileNumber);
dataRepetitions = 0;

Please note that creating instance of FooterType for each file may be superfluous. It could be created outside loop, for example at 60 line

Thanks for the response, still working on it but was helpful! — Tommy Hamm, Mar 30 '17 at 14:49

Splitting Large XML FIles Java (StAX)

1 Answers1