0

I am working with a gigantic loop in a static void main(String[] args). This loop manipulates some numeric primitives that I want to get into a CSV, where each row in the CSV corresponds to an iteration in the loop.

My objective is to dynamically write these primitives such that at the end of each iteration they can be garbage collected and essentially forgotten about. The worst thing would be to have these stored in memory until the end of the loop, since the loop is very long.

I have written a class that attempts to do this, which is pasted below. Question: Is each row of the CSV being stored into memory and then written to disk at the end of the loop? If so, how do I make it such that disk writing happens at every loop iteration to free up memory (preferably in a way that is fast)?

public static void main(String[] args) throws Exception {
    WriteCSV csvWriter = new WriteCSV("src","Hello.csv")

    for(int i = 0 ; i < 1000 ; ++i) { //Much bigger in real-world case
        csvWriter.writeRow(i);
    }
    csvWriter.close(); //Does all i between {1,2,...,1000} get GC'd here or dynamically in the above loop???
}

CSV writing class to write dynamically in a loop:

class WriteCSV {
    private FileWriter fstream;
    private BufferedWriter out;


    public WriteCSV(String directory, String filename) throws IOException {     
        File file = new File(directory, filename);

        this.fstream = new FileWriter(file);
        this.out = new BufferedWriter(this.fstream);
    }


    public void writeRow(int newRow) throws IOException {
        this.out.write(String.valueOf(newRow));
        this.out.newLine();
    }

    public void close() throws IOException {
        this.out.close();
    }

}
user2763361
  • 3,789
  • 11
  • 45
  • 81

2 Answers2

1

Map the file to the memory and use MemoryMappedBuffer. You won't beat the OS (most probably). Guava has a handy shortcut for this.

bobah
  • 18,364
  • 2
  • 37
  • 70
  • So are you saying that it *is* currently storing **all* the `int`s to memory until I call `close()`? – user2763361 Oct 21 '13 at 10:08
  • It allows you to work with the file as with memory buffer. It will flush it automatically (or you may call buffer.force() if to force it, discussion here: http://stackoverflow.com/questions/4096881/do-we-need-to-use-mappedbytebuffer-force-to-flush-data-to-disk) – bobah Oct 21 '13 at 10:16
  • Perhaps you can give an example? – user2763361 Oct 21 '13 at 12:04
0

As you're using BufferedWriter, it may not write to file at the end of loop. This depends on your data size. BufferedWriter uses 8192 as default size to write to file, which means it will not write to file unless your data size cached by the writer is up to 8192.

criszhao
  • 134
  • 6