2

Having an odd problem - one of my app suites has to read/write gzip-compressed files that are used on both Windows and Linux, and I am finding that the files I generate using zlib on Linux are 2-3 times larger than those I generate using GZipStream on Windows. They read perfectly on either platform, so I know that the compression is correct regardless of which platform created the file. The thing is, the files are transferred across the network at various times, and obviously file size is a concern.

My question is:

  1. Has anyone else encountered this
  2. Is there some documented difference between the two? I do know that GZipStream does not provide a way to specify the compression level like you can with zlib, but I am using maximum compression on the zlib side. Shouldn't I see relatively the same file size, assuming that GZipStream is written to use maximum compression as well?
Adi Lester
  • 24,731
  • 12
  • 95
  • 110
Will
  • 3,500
  • 4
  • 30
  • 38
  • Just to clarify, what is being compressed is (essentially) a single, monolithic file, making use of the gzip format using deflateInit2() with Z_BEST_COMPRESSION, Z_DEFLATED, 16+MAX_WBITS, and Z_DEFAULT_STRATEGY. – Will Jun 30 '11 at 23:00
  • You cannot specify the compression level for GZipStream. It sure doesn't sound like you want to if they are 3 times smaller :) – Hans Passant Jun 30 '11 at 23:34
  • Exactly ... the problem is on the Linux side. Been racking my brain over this one for a while and finally decided to throw this one out to the SO gurus and see if they had any ideas. – Will Jul 01 '11 at 11:42
  • can you show fragments of code, the pertinents ones ? – woliveirajr Jul 01 '11 at 14:22

2 Answers2

1

I think the reason you are experiencing this is not because of the compression algorithm used, but because of how the files are compressed. From the zLib manual:

"The zlib format was designed to be compact and fast for use in memory and on communications channels. The gzip format was designed for single- file compression on file systems, has a larger header than zlib to maintain directory information, and uses a different, slower check method than zlib."

I think what is happening is that the files on your linux machine are being Tar'red together into 1 file, then that one file is being compressed. In WIndows, I think it compresses each individual file, then stores them compressed into 1 file.

This is my theory, but have nothing to really support it. Thought I might try some trial tests at home later, just to fill my curiosity.

Ray Hayes
  • 14,896
  • 8
  • 53
  • 78
Icemanind
  • 47,519
  • 50
  • 171
  • 296
  • I would lean towards that, except I know it's only one file being compressed. I have a proprietary archive format that I am using; the archive is generated, the entire archive is compressed as one file, and then the result is encrypted. Same process on both platforms. – Will Jun 30 '11 at 22:53
  • @Will - So much for my theory then. I will run some tests and see if I can't figure out what is going on. I have Linux on a VMWare virtual machine and so I will see if I can't figure out what's going on and change my answer. – Icemanind Jul 01 '11 at 03:14
  • thanks for your help. I've been trying various combinations, and nothing comes close to the results I get on the .NET side. Very odd to me, especially knowing how long zlib has been around. – Will Jul 01 '11 at 11:43
1

And the answer is .... the Linux version was never compressing the data to begin with. Took a lot of debugging to find the bug that caused it, but after correcting it, the sizes are now comparable on both platforms.

Will
  • 3,500
  • 4
  • 30
  • 38