OutOfMemoryException when creating large ZIP file using System.IO.Packaging

Question

I am trying to debug an OutOfMemoryException that occurs when creating a fairly large ZIP file using System.IO.Packaging.ZipPackage.

The code is iterating through a large list of objects, doing the following for each object.

Serializing the object data to a temporary file.
Creating a PackagePart for the file.
Copy from a source System.IO.Stream to another:
- Source stream: FileStream
- Target stream: PackagePart::GetStream() => MS.Internal.IO.Zip.ZipIOModeEnforcingStream

Finally it calls Package::Close() which saves the file.

The problem I am having is that for a particularly large list of objects, I am seeing an OutOfMemoryException (the x86 process size is getting to about 1.2GB in size).

I was thinking about partitioning the object data into chunks so I only process a smaller amount per loop (i.e. steps 1-3 above). The idea is that I would create n ZIP files in a temporary directory, and then find a way to combine them into a single archive.

Is this possible using System.IO.Packaging? What would I use to combine the parts?

Or is there a better way to fix this?

Try calling Flush in between the iteration when creating a new package — Ankur, Oct 21 '11 at 06:40
@Ankur - you've nailed it, that seemed to do the trick. If you upgrade your comment to an answer, I will be happy to accept it. — LeopardSkinPillBoxHat, Oct 24 '11 at 00:35

score 3 · Accepted Answer · answered Oct 24 '11 at 04:15

3

Calling the Flush method on the Package object in between creating a new package should probably solve the problem as that would cause the memory buffer to be flushed to disk.

answered Oct 24 '11 at 04:15

Ankur

33,367
2
46
72

Thanks - this worked for me. I ended up flushing it between every 500 files that were added to the ZIP file. This was a good tradeoff between memory and performance. – LeopardSkinPillBoxHat Oct 24 '11 at 04:32

score 0 · Answer 2 · answered Oct 21 '11 at 03:50

0

I would use the DotNetZip library (http://dotnetzip.codeplex.com/). I've tried several zip libraries (System.IO like you are currently using and also SharpZibLib) and by far the easiest to use is the DotNetZip library.

You'll almost certainly end up with fewer lines of code and I found the memory usage to be very good (had a problem in a virtual machine environment which I reported and a new release fixed it).

answered Oct 21 '11 at 03:50

Ben Robbins

2,889
2
31
32

My original question wasn't very clear about this, but this is actually a defect in existing code and I didn't want to have to re-write this code too much (I just want to fix the crash). I edited the question to make this more clear. – LeopardSkinPillBoxHat Oct 21 '11 at 03:56

OutOfMemoryException when creating large ZIP file using System.IO.Packaging

2 Answers2

Linked