How to append a child element to a large XML file using Java?

Question

I am trying to create a XML file, using Java, that is a collection of GPS coordinates (GPX). Every time I receive a coordinate from my android device (approximately 1 every second) I need to append the results to an existing XML file. The output that I am looking for is shown below with the trkpt element as the repeated item. The problem is that I can't just add the new trkpt to the end of the file because it needs to be inside the trkseg parent element.

So far I have tried two different APIs, SIMPLEXML and JDOM. With SIMPLEXML I couldn't figure out how to append a child element to an existing file so I switched to JDOM. JDOM allowed me to append the trkpt element as shown below, but as the file started growing it quickly slowed down the user interface of the program. With JDOM I was using the SAXBuilder to reopen the file and append. I think the issue with this was that it had to reproduce the entire file in memory before it added the new element and rewrote the file. So the larger the file got the more demanding the operation was on the device. I need a solution that doesn't examine/copy the entire file before writing the new data. Is there a more efficient way to accomplish this with Java or an API for Java? Thanks for any help!

<?xml version="1.0" encoding="UTF-8"?>
<gpx xmlns="http://www.topografix.com/GPX/1/1">
        <trk>
            <trkseg>
                <trkpt lon="9.860624216140083" lat="54.9328621088893">
                    <ele>228.0</ele>
                </trkpt>
                <trkpt lon="9.860624216140100" lat="54.9328621088754">
                    <ele>234.0</ele>
                </trkpt>
                <trkpt lon="9.860624216140343" lat="54.9328621088678">
                    <ele>227.0</ele>
                </trkpt>
            </trkseg>
        </trk>
</gpx>

Are you always appending data in the same way? Then consider just reading the file line by line and append your line in a 'XML-agnostic' way; as long as the result is still valid XML. — The Nail, Feb 18 '12 at 17:53
Are you to able to instead append the data to a plain file and convert the whole file to XML later? Or maybe convert it in batches? Trying to update XML continuously seems like making life unnecessarily difficult for yourself! ;-) — DNA, Feb 18 '12 at 17:55
I suppose your question relates to client-side processing (android) and not server-side processing, correct? — home, Feb 18 '12 at 18:01
@The Nail, I am currently appending the data in the same way every time I edit the file. — Ferrari692, Feb 18 '12 at 18:54
@DNA I like your suggestion. I will be using the data later with my server and could process it into XML at that time. If all else fails that is probably the route I will take. — Ferrari692, Feb 18 '12 at 18:55
@home my question is dealing with client-side processing on an android device. However, like DNA suggested I could look into doing server-side processing. — Ferrari692, Feb 18 '12 at 18:58
@Ferrari692 - What did you end up doing? What is working for you? Please post your working solution, thanks. — snahl, Aug 27 '14 at 19:43

score 0 · Answer 1 · answered Feb 18 '12 at 18:02

There are always bottlenecks when it comes to I/O, especially when opening/closing/re-opening files in a repetative way.

A DOM handler will create a whole tree structure every time it opens the file, but is very effective when it comes to alter that tree.

So first of all, do you really need to open, alter, save the file on every tick? If not, keep the DOM of the file in memory, and alter through the reference to the XML. Save when the user exits the app or leaves a view.

If you do need to save the file at each tick, you could still keep the DOM in memory and only save it to disk on each tick.

If you need to open/save/re-open the file on each tick, don't use any XML-library - simply use a standard FileWriter or such alike, with manual altering of the contents - but it will still be hard to maintain performance if the file gets really large.

I don't necessarily need to save to the file on every tick. As long as there isn't a greater chance of loosing the data while it is waiting to be written. I guess I should mention that the amount of data could get into the GB range so saving when exiting the app might be a problem. — Ferrari692, Feb 18 '12 at 18:44
@Ferrari692 Yeah, and you shouldn't keep a DOM object of that size in memory as well. Consider using other methods than parsing a whole file of that size, either chunks or non-XML. — Björn, Feb 18 '12 at 18:46

score 0 · Answer 2 · answered Feb 18 '12 at 18:12

This sounds like the perfect application for SAX (find it in the package org.xml.sax); it’s a streaming API for XML access and manipulation. SAX generates events for every element it encounters, allowing you to copy the file to a new file without having to parse it into a large in-memory tree. When you have reached the end of your input file, just append the new element as appropriate before processig the end tag of <trkseg>.

Of course your approach of rewriting this file every second sounds questionable in and of itself. Can you bundle the information in larger segments? You could dump the information to single files and collect them into a single file at certain intervals (every 10/30/60 seconds).

I wouldn't have an issue bundling the info and using larger intervals. I will look into using SAX. — Ferrari692, Feb 18 '12 at 18:49

score 0 · Answer 3 · answered Feb 18 '12 at 18:46

0

If it's as simple as stated here you can just use a RandomAccessFile and seek to the file length minus a few bytes (right before the root closing tag) and just start overwriting there.

answered Feb 18 '12 at 18:46

Vincent Vanmarsenille

494
2
5

Interesting. I will have to try that out. – Ferrari692 Feb 18 '12 at 19:00

score 0 · Answer 4 · answered Feb 19 '12 at 06:39

I suggest to split the xml file into 3 parts.

head.xml

<?xml version="1.0" encoding="UTF-8"?>
<gpx xmlns="http://www.topografix.com/GPX/1/1">
    <trk>
        <trkseg>

body.xml

<trkpt lon="9.860624216140083" lat="54.9328621088893">
    <ele>228.0</ele>
</trkpt>
<trkpt lon="9.860624216140100" lat="54.9328621088754">
    <ele>234.0</ele>
</trkpt>
<trkpt lon="9.860624216140343" lat="54.9328621088678">
    <ele>227.0</ele>
</trkpt>

tail.xml

        </trkseg>
    </trk>
</gpx>

now whenever you get new data, simply append it to body.xml

to read xml file use SequenceInputStream as below:

List<InputStream> list = new ArrayList<InputStream>(3);
list.add(new FileInputStream("head.xml"));
list.add(new FileInputStream("body.xml"));
list.add(new FileInputStream("tail.xml"));
InputStream xmlStream = new SequentialInputStream(Collections.enumeration(list));

How to append a child element to a large XML file using Java?

4 Answers4