0

I've run into a bit of an issue with gzip compression.

Say we compress the letter 'a' with pako library in javascript, as follows:

pako.gzip('a')

This returns the following byte array:

[31,139,8,0,0,0,0,0,0,3,75,4,0,67,190,183,232,1,0,0,0]

Now, if I take this array and decompress it in python, I get the result as expected:

import gzip
arr_from_pako = bytearray([31,139,8,0,0,0,0,0,0,3,75,4,0,67,190,183,232,1,0,0,0])
decompressed = gzip.decompress(arr_from_pako)
print(decompressed)
>>> b'a'

However, if I then reverse the process and gzip compress the result in python, I get a different bytearray:

arr_from_python = list(gzip.compress(decompressed))
print(arr_from_python)
>>> [31, 139, 8, 0, 123, 15, 138, 98, 2, 255, 75, 4, 0, 67, 190, 183, 232, 1, 0, 0, 0]

What I need to do here is reproduce the result of pako.gzip in python. I'm guessing the differences are due to a compression level and I know gzip lib in python lets you adjust it but I tried every single setting and was not able to get the expected result.

Can anyone help?

itsk
  • 9
  • 1

1 Answers1

0

You can't, and you don't need to. There is no issue.

  1. The difference in the examples you give is only in the gzip header, where Python is putting in a modification time and setting some other bytes in the header to different values than pako. You can use mtime=0 in Python to get rid of the modification time, but you will still have those other differences. (0, 3 vs. 2, 255 — those are the XFL and OS flags in the header.)

  2. You could manually rewrite the header in Python to be the same as gzip. However that still won't meet your objective in the long run, once you are compressing reasonable amounts of data. Then, even if you force the headers to be the same, the compressed data can and likely will be different. The only way to assure that the compressed data is the same is for you to be personally in control of the libraries used in both Python and pako, make sure that they are the same source code, the same version of that source code, and are using the exact same controlling parameters. I doubt that you are, and so you are on a fool's errand to try to make them the same.

  3. There is no need to make them the same. All you need from a lossless compressor is compression followed by decompression to give you exactly what you put in. That's it. You have that. There is never any assurance of the other direction, i.e., that you can get decompression followed by compression to give you the same thing.

Mark Adler
  • 101,978
  • 13
  • 118
  • 158
  • Thanks for the clarification, it makes a lot of sense now. In my particular use case I actually need the results to match as the resulting byte array is then hashed and I need this hash to be identical. I've implemented a workaround though. Thanks once again. – itsk May 24 '22 at 12:47
  • You could use the CRC at the end of the gzip file as your hash. That is a CRC of the uncompressed data, which is what matters. – Mark Adler May 24 '22 at 14:25
  • Yes, it would make much more sense to use a CRC checksum here. Unfortunately, I can't modify the JS bit and it has been designed by someone else. – itsk May 24 '22 at 15:03