Spring-Hibernate How to do paginated marshal with JAXB xml bind

Question

There is a really huge data set that being expected to be inserted into an xml. I decided to use jaxb.xml.bind.Marshaller.

Because of that huge data, I have to select that data from db by pagination.

I mean, for instance, I have to get 10.000 row per each process from db and immediately write to xml, and this process continue up till the 3million rows will be written.

The most crucial restriction is, I have to use only one xml file. In other words, I have to marshall this huge data into only one xml file. Therefore, I would like to know if it is possible marshal data into one xml, and then reuse and update the same xml bu marshalling again.

Thanks a lot.

Answer 1

You can actually do it in one go, but with less convenient frameworks. First of all go for raw JDBC (probably with JdbcTemplate, see methods taking RowCallbackHandler) and instead of mapping the whole table (3 million rows) into a huge collection, iterate over the result (e.g. using ResultSet). This way you can avoid several paginated queries and have minimal memory footprint (even smaller compared to fetching 10000 rows at a time).

Outputting such a big XML file is a bigger problem. sax can't be used for output and dom is not suited for such a big files. I think the best approach is to print XML directly using PrintWriter...