How to write bean to CSV using OpenCSV without converting to String[]?

Question

I have a large array of a bean that needs to be written to a CSV file. I am trying to use OpenCSV but all the examples I see on the web ask me to convert the bean (POJO) that I have to a string array. I don't want to do this since this would mean I will have to go through the million beans and convert each of those beans to String[] which would be a very memory-expensive process.

This is the piece of code I have right now for doing this:

private static void writeRowsToCsv(Path filePath, List<MyBean> rows)
        throws IOException {

    StringWriter writer = new StringWriter();
    CSVWriter csvWriter = new CSVWriter(writer, '#', '\'');
    ColumnPositionMappingStrategy mappingStrategy =
            new ColumnPositionMappingStrategy();
    mappingStrategy.setType(MyBean.class);
    List<String[]> rowsInStringArray = convertBeanToStringArray(rows)
    csvWriter.writeAll(rowsInStringArray);
}

Is there a way I can avoid the conversion to String[] and use the list of beans that I have to write to a CSV file?

Why would that be very memory-expensive? You call `writeNext()` in a loop, converting one bean at a time. — Andreas, Apr 10 '17 at 14:51
That would mean I am writing one row at a time. I have a millions of rows to be written, which would make it extremely slow. — jobin, Apr 10 '17 at 14:52
What do you think `writeAll()` does? It iterates the list and writes them, one at a time. How else would it process the list? — Andreas, Apr 10 '17 at 14:52
I haven't investigated but I believe it writes a batch of rows at a time. — jobin, Apr 10 '17 at 15:04
What batching? There is no batching. Stop trying to guess when you have no clue. `writeAll()` is nothing but a loop calling `writeNext()`. See [source code](https://sourceforge.net/p/opencsv/source/ci/master/tree/src/main/java/com/opencsv/CSVWriter.java#l178). — Andreas, Apr 10 '17 at 15:09
Thanks @Andreas. That means the best way out here is to call `writeNext()` in a loop. If you can write that out as an answer, I could mark it accepted. — jobin, Apr 10 '17 at 15:17

score 8 · Answer 1 · edited Oct 31 '17 at 07:22

8

Jobin, seriously, if you are using OpenCSV then use the BeanToCsv or the newer StatefulBeanToCsv. It will save your sanity.

private static void writeRowsToCsv(Path filePath, List<MyBean> rows)
        throws IOException {

    StringWriter writer = new StringWriter();
    ColumnPositionMappingStrategy mappingStrategy =
            new ColumnPositionMappingStrategy();
    mappingStrategy.setType(MyBean.class);

    StatefulBeanToCsvBuilder<MyBean> builder = new StatefulBeanToCsvBuilder(writer);
    StatefulBeanToCsv beanWriter = builder
              .withMappingStrategy(mappingStrategy)
              .withSeparator('#')
              .withQuotechar('\'')
              .build();

    beanWriter.write(rows);
}

edited Oct 31 '17 at 07:22

Garis M Suero

7,974
7
45
68

answered May 08 '17 at 20:31

Scott Conway

975
7
13

2

The constructor StatefulBeanToCsvBuilder(CSVWriter) does not exist. – ledlogic Oct 24 '17 at 23:35
2

ledlogic - you are absolutely correct. I corrected my code example above as the StatefulBeanToCsv creates a CSVWriter based on values passed in from the builder but it does not take a CSVWriter directly. Not exactly sure what I was thinking back in May but thank you for pointing that out because who knows how long that would have stayed there. – Scott Conway Oct 25 '17 at 12:19
this is the easier way to do it. – Bharath Oct 30 '17 at 22:35
In OpenCSV 5.7.0, there is a constructor StatefulBeanToCsvBuilder(ICSVWriter) and CSVWriter implements it via AbstractCSVWriter. – pyb Oct 18 '22 at 19:41

score 1 · Accepted Answer · answered Apr 10 '17 at 15:18

1

You call writeNext() in a loop, converting one bean at a time. Not memory-expensive at all.

answered Apr 10 '17 at 15:18

Andreas

154,647
11
152
247

How to write bean to CSV using OpenCSV without converting to String[]?

2 Answers2