Creating large multi-part zip with Ionic fails with OutOfMemoryException

Question

I try to create a multi-part zip file with files of a total size of 17 GB, using IonicZip.

Each zip part is set to be not larger than around 500 MB.

zip.MaxOutputSegmentSize = 500000000;

The files that go into the zip are of various sizes each, but never larger than 350MB, usually much smaller, just a couple of KB or MB.

My machine where I create the zip file has 4GB RAM.

When I start the zip creation in my program, I get an OutOfMemoryException at some point due to the used RAM.

(The same works fine when the total size of all files is about 2 GB instead of 17GB).

The code:

ZipFile zip = new ZipFile(zipFileName, Encoding.UTF8);
zip.CompressionLevel = CompressionLevel.BestCompression;

zip.ParallelDeflateThreshold = -1; // https://stackoverflow.com/questions/11981143/compression-fails-when-using-ionic-zip
zip.MaxOutputSegmentSize = 500000000; // around 500MB

[...]

while (...)
{
  zip.AddEntry(adjustedFilePath, File.ReadAllBytes(filepath));
}

zip.Save();

I am wondering how IonicZip handles zip.save in combination with multi-part creation. It should not be necessary to hold all multi-parts in memory but only the current one, not?

And since I set zip.MaxOutputSegmentSize to only around 500MB and the maximum size of a single file that goes into the zip is never more than 350MB, I don't see why it should eat up so much memory.

On the other hand, when the OutOfMemoryException occurs, there is not even any single part of the multi-part written to disk yet. Usually, with smaller amount of files where the zip creation succeeds, the multiple parts are on the filesystem with different creation timestamps, approx. 5 seconds apart each. So I am really not sure what IonicZip is doing internally exactly until it spits out the first zip part.

Sorry, I'm new to C# and .NET. Is the IonicZip the best library for this? Could I use the System.IO.Compression or System.IO.Packaging package instead (I did not see that they support multi-part zips) or the commercial Xceed?

Posts that I already checked but did not help:

Compression fails when using ionic zip
Ionic zip throws out of memory exception
OutOfMemoryException when creating large ZIP file using System.IO.Packaging (not IonicZip related)

score 0 · Answer 1 · answered Aug 19 '16 at 07:35

0

I suggest using the built in .net System.IO.Compression namespace classes to zip/unzip files; they are stream based, whereas your example here does not use streams - so must be using RAM to store the compressed data. With the .net native libraries you can write out compressed data in chunks to the output stream.

answered Aug 19 '16 at 07:35

PhillipH

6,182
1
15
25

Thanks. Does this package support multi-part zips, cause I could not find anything about it in Google or https://msdn.microsoft.com/de-de/library/system.io.compression.zipfile(v=vs.110).aspx – Mathias Conradt Aug 19 '16 at 08:09
Zip does, but the others (GZip) do not. – PhillipH Aug 19 '16 at 08:50
Could you point me to some online API docs where the split size can be set? I still cannot see where I can do that via the `System.IO.Compression` package. Right now I'm testing Xceed (https://xceed.com/xceed-zip-for-net/) library as an alternative, where I can simply say `zip.splitSize= 500000000`. So far I cannot see any increase in memory usage but it"s still zipping up my 17GB. – Mathias Conradt Aug 19 '16 at 09:35
Sorry - I don't know what SplitSize is or what its used for. – PhillipH Aug 19 '16 at 11:21
To create a multipart zip file. Since I need to zip 17GB of files, I don't want to have one large zip file, but split into multiple, so in the end I will have like myfiles.zip, myfiles.z01, myfiles.z01, etc. each not being larger than 500MB. Especially since the zip format has a limit of 4GB per file. "SplitSize" is just the name of the paramter as Xceed uses to define the max size of such a part. In IonicZip it"s called "MaxOutputSegmentSize". – Mathias Conradt Aug 19 '16 at 11:56
Then I don't think the Compression namespace libraries support multi file zip archives natively - however since the input and output streams can be controlled by the programmer, theres no reason you couldn't change output stream halfway through the compression operation and so effectively write to several files. – PhillipH Aug 19 '16 at 12:39

Mathias Conradt · Accepted Answer · 2016-08-20T07:19:48.987

The solution I came up with which seems to be the best in terms of effort (I don't want to having to reinvent the wheel and write multi-part logic for the standard zip/compression package, seems such a standard thing that should be part of any zipping package) is to use Xceed and pretty much use the same code logic like I did for IonicZip.

It seems that IonicZip is not really handling it in an efficient way in terms of memory. Xceed works fine without any major increase in memory usage.

        using Xceed.Zip;
        using Xceed.FileSystem;

        [...]

        ZipArchive zip = new ZipArchive(new DiskFile(zipFileName));
        zip.SplitNameFormat = SplitNameFormat.PkZip;
        zip.SplitSize = 500000;
        zip.DefaultCompressionLevel = Xceed.Compression.CompressionLevel.Highest;
        zip.TempFolder = new DiskFolder(@"C:\temp");
        new DiskFolder(localSourceFolder).CopyFilesTo(zip, true, true);

Creating large multi-part zip with Ionic fails with OutOfMemoryException

2 Answers2