0

In the existing project user can download report and it has 10 million records, this process gets data from database and writes to csv by using super csv java api then sends an email to user by attaching, it takes huge heap space to hold 10 million java objects and writing these records to csv files, because of this server is crashing and going down as application has many reports like this. is there any better way to handle this.? I red sxssfworkbook documentation and it says specified records count can keep in memory and remaining records will be pushed to hard disk but this is using to create excel files. is there any similar api to create csv files or sxssfworkbook can be used to create csv files.?

Cœur
  • 37,241
  • 25
  • 195
  • 267
Ramesh
  • 45
  • 1
  • 9
  • 1
    Did you try to increase the -Xmx-parameter? Can you buy more RAM? – user unknown Apr 08 '18 at 13:11
  • -Xmx params are good for usual scenario's in application, only problem is with reports which has 10 millions records, infrastructure team will not agree to increase memory. – Ramesh Apr 08 '18 at 13:26
  • I'm not sure I understood your processes. Maybe you can compress the data? CSV-files can often be compressed dramatically. – user unknown Apr 08 '18 at 16:51

1 Answers1

2

There are few Java libraries for reading and writing CSV files. They typically support "streaming", so they do not have the problem of needing to hold the source data or the generated CSV in memory.

The Apache Commons CSV library would be a good place to start. Here is the User Guide. It supports various of flavors of CSV file, including the CSV formats generated by Microsoft Excel.

However, I would suggest that sending a CVS file containing 10 million records (say 1GB uncompressed data) is not going to make you popular with the people who run your users' email servers! Files that size should be made available via an web or file transfer service.

Stephen C
  • 698,415
  • 94
  • 811
  • 1,216
  • Thank you Stephen, I will play with Apache commons csv api and I will consider to use file transfer service instead of email. – Ramesh Apr 09 '18 at 01:05