-2

we were testing the capability of our application, here is the scenario, we have a table that holds 5 million record, we want to export it to csv so we successfully made a csv writer class. the main problem is when we export the whole record, we catch OOM exception(out of memory exception), when the data is over a million, what makes it more difficult is the user can choose what columns are to be exported.. is there any way to catch thing up nice and well without a blocking process??? what type of process is best suitable for this scenario??? should i used (and how) to use background process for this??

any point of view (examples) are well appreciated. thanks...

ECie
  • 1,265
  • 4
  • 21
  • 50

2 Answers2

5

Well it's difficult to answer with such a short description. Based on your tags; it seems it's a ASP.NET MVC4 based website. Also it seems; you are trying to create the CSV on-the-fly. (i.e. within the single http call) I personally would prefer an AJAX based solution where the sequence will be as follows.

  1. User initiates the export.
  2. Server receives the export request and starts a background job to create the CSV file. (possibly in a different process) and return the status saying job started (With Job ID).
  3. Implement a mechanism to monitor this job. Possible a DB entry about the job status and progress.
  4. At front-end a periodic AJAX call checks the job status; and shows progress to the user.
  5. Once the job finishes; provide a download link for user to download your huge file.

With proper implementation; this approach has the following benefits.

  • It's Non blocking and interactive.
  • It can handle really huge jobs.
  • User gets an opportunity to cancel the job.
  • Each job runs in a separate process; giving it it's own memory space.
  • It can be scaled to multiple servers.

Although as you can see; it has it's own overheads.

Malay Sarkar
  • 411
  • 3
  • 9
  • exactly sir @malay but i dont have enough knowledge on that background job, i am currently looking at capability of signal r hence it is a long pooling but with the background work i dont have a starting point yet – ECie Nov 12 '14 at 06:50
  • i am also looking to a web service sir for a possible approach – ECie Nov 12 '14 at 06:54
0

use file helpers library see article

@ FileHelpers throws OutOfMemoryException when parsing large csv file

you can get it via nuget

Community
  • 1
  • 1
GuyP
  • 1
  • This is a borderline [link-only answer](http://meta.stackexchange.com/q/8231/213671). You should expand your answer to include as much information here, and use the link only for reference. – gunr2171 Jan 08 '15 at 20:10