adding Elements to Serialized ArrayList without loading ArrayList into RAM in JAVA

Question

Is it possible to add Elements into an Array list in java that is already serialized to the disk without reloading it into RAM. I Need this for saving strucutred data from an XML file to load data into a mysql database. The XML File has several GB of Data and the Problem is that i have to store all of the data from the XML file to process the data?

Is the ArrayList serialized using standard Java serialization? An alternative serialization? Your own serialization? Do you have to load the XML file as the original Java objects, or can you just process it using standard XML tools? — RealSkeptic, Feb 06 '17 at 08:07
I want to serialize it with Standard Java serializazion. I'm proccessing the XML file row by row, because it is to big to load it into ram. I Analyse every tag and store it in Arrays, i want to serialize these Arrays into objects on disk if the size in ram exceed some value — arget888, Feb 06 '17 at 08:11

score 0 · Accepted Answer · answered Feb 06 '17 at 08:09

0

ArrayList, or any other data structure for that matter, is an in-memory data structure. So, whatever you wish to put in this data structure HAS to be loaded into memory(RAM).

If you're wondering how to load data that is more than the memory size itself, consider loading in batches. e.g. Load the first n entries which fit into the memory constraint and then process them. Once you're done, discard those and load the next batch and so on.

answered Feb 06 '17 at 08:09

MuchMore

168
1
4

OK i thouht that i had to go this way, but i hoped that someone knows a possibility to do that in a different way – arget888 Feb 06 '17 at 08:32

score 0 · Answer 2 · answered Feb 06 '17 at 08:18

I think since your XML file is already in GBs then better approach would be to read XML file in Chunks and Save Chunk of Records to Database. Keep doing it until EOF reached.

String encoding = "UTF-8";
int maxlines = 100;
BufferedReader reader = null;
BufferedWriter writer = null;

try {
    reader = new BufferedReader(new InputStreamReader(new FileInputStream("/bigfile.txt"), encoding));
    int count = 0;
    for (String line; (line = reader.readLine()) != null;) {
        if (count++ % maxlines == 0) {
            close(writer);
            writer = new BufferedWriter(new OutputStreamWriter(new FileOutputStream("/smallfile" + (count / maxlines) + ".txt"), encoding));
        }
        writer.write(line);
        writer.newLine();
    }
} finally {
    close(writer);
    close(reader);
}

adding Elements to Serialized ArrayList without loading ArrayList into RAM in JAVA

2 Answers2