Compressing C# objects using SharpZipLib?

Question

Is it possible to compress a List<T> using SharpZipLib? Turning the List to a byte array gives me around 60000 bytes (uncompressed).

Compression this with System.IO.Compression.DeflateStream gives me around a 1/3 compression rate but this is far from enough.

The purpose is to store the collections in the (MS SQL) database as a byte[] because saving them as individual rows uses to much space (1 million rows/day).

Thanks

Edit:

List<ItemLog> itemLogs = new List<ItemLog>();
//populate with 1000 ItemLogs
byte[] array = null; //original byte array

BinaryFormatter bf = new BinaryFormatter();
MemoryStream ms = new MemoryStream();
bf.Serialize(ms, itemLogs);
array = ms.ToArray();

the array size is now 60000 bytes

Zip the collection using a ZipOutputStream

MemoryStream outputMemoryStream = new MemoryStream();
ZipOutputStream zipStream = new ZipOutputStream(outputMemoryStream);

zipStream.SetLevel(3);

ZipEntry entry = new ZipEntry("logs");
entry.DateTime = DateTime.Now;

zipStream.PutNextEntry(entry);

StreamUtils.Copy(ms, zipStream, new byte[4096]);
zipStream.CloseEntry();

zipStream.IsStreamOwner = false;
zipStream.Close();

outputMemoryStream.Position = 0;

byte[] compressed = outputMemoryStream.ToArray();

The compressed is now 164 bytes in size. <- length not valid/possible?

Uncompressing gives me a empty array. But as the compression is not right I will skip the uncompression code for now.

@Julo I did this. The original [] size is 60000 and the library compresses it to 164 bytes. Will edit with code in a few minutes — lordvlad30, Mar 27 '18 at 08:43
This does not seem to be valid length. Post your code to check the reason for the small size. — Julo, Mar 27 '18 at 08:43
For now, the line `StreamUtils.Copy(ms, zipStream, new byte[4096]);` seems to be wrong. I need to check it with the real implementation of SharpZipLib how this code will work, but the `new byte[]` seems out of place. — Julo, Mar 27 '18 at 08:54
@Julo I also tried setting the buffer to the actual array size (60000 bytes) but the result was the same. — lordvlad30, Mar 27 '18 at 08:55

Julo · Accepted Answer · 2018-03-27T10:40:49.170

I do not see any real problem in your code. The only part, where the problem can be is copying of data. Is the stream of input data at the start of data, that should be stored? Try add the following line:

ms.Seek(0, SeekOrigin.Begin); // added line
StreamUtils.Copy(ms, zipStream, new byte[4096]);

Based on your code I wrote a simple compress function and it works as expected.

private static byte[] Compress(byte[] source)
{
  byte[] compressed;
  using (var memory = new MemoryStream())
  using (var zipped = new ZipOutputStream(memory))
  {
    zipped.IsStreamOwner = false;
    zipped.SetLevel(9);
    var entry = new ZipEntry("data")
    {
      DateTime = DateTime.Now
    };
    zipped.PutNextEntry(entry);
#if true
    zipped.Write(source, 0, source.Length);
#else
    using (var src = new MemoryStream(source))
    {
      StreamUtils.Copy(src, zipped, new byte[4096]);
    }
#endif
    zipped.Close();
    compressed = memory.ToArray();
  }
#if false
  using (var file = new FileStream("test.zip", FileMode.Create, FileAccess.Write, FileShare.Read))
  {
    file.Write(compressed, 0, compressed.Length);
  }
#endif
  return compressed;
}

You have there alternatives how to save the output (array or stream) and there is disabled code to save compressed data to file (to check in a external application the real content of compressed data).

My tested data were a 256 bytes long (data, with small compression rate), and the result was 407 bytes (file).

Try to use array, or check the stream content that is saved.

Will test this but I never save the compressed data as a Zip file, always as a byte array. — lordvlad30, Mar 27 '18 at 09:57
SharpZipLib creates data, that are a real ZIP file. Therefore there will be a bigger 'header', but the data will be accessible even without a special application for decompression. — Julo, Mar 27 '18 at 10:01
I will accept this answer. The compression at the end is about the same as the `System.IO.Compression.Deflate`. — lordvlad30, Mar 27 '18 at 11:01
Was it the `Seek` that was missing? Or did you change to use it with byte array? The size should he similar, because standard ZIP method is Deflate. I'm not sure, whether you can change the compression method in SharpZipLib, but ZIP format now supports also advanced methods. — Julo, Mar 27 '18 at 11:33
@Juno I actually changed the `StreamUtuils.Copy` to a `zipDestination.Write(source, 0, source.Length)` (or something like that, don't have access to the code right now) — lordvlad30, Mar 27 '18 at 11:42

Compressing C# objects using SharpZipLib?

1 Answers1