0

I'm trying to decompress a block of memory but I don't know it's original uncompressed size, however, I do know the maximum size this original block of memory could ever be.

Is there any way to decompress with LZ4 without knowing the original uncompressed size?

Short-Story-Long: I'm serializing a bunch of variables into a stringstream using a third party library called Cereal. Cereal will serialize your data into a stringstream using a 'portable binary format' which means the endianness is preserved and even converted to that of the host machines during deserialization if needed. The stringstream is then compressed using LZ4 and transmitted to a remote machine for decompression and deserialization.

My issue is that LZ4 will output a memory block of compressed data and an integer specifying the compressed data's size. To decompress you need the compressed size and original size and I'm only sending over the compressed data block to the remote machines.

SO, is there any way to decompress a block of data with LZ4 without knowing it's original compressed size? Essentially 'start decompression, when you run out of data you're done'

KKlouzal
  • 732
  • 9
  • 32

1 Answers1

1

The function LZ4_decompress_safe (see https://github.com/Cyan4973/lz4/blob/master/lib/lz4.c#L1288) seems to require only the maximal decompressed size. In this case, you can allocate a buffer large enough for decompression and use it.

So either that, or transfer the original uncompressed size to the remote machines as well.

EDIT: In your case, you can also use LZ4 streaming decompression, see this code sample for more information.

Tal Ben-Nun
  • 439
  • 3
  • 7
  • `LZ4_decompress_safe(const char* source, char* dest, int compressedSize, int maxDecompressedSize)` requires compressedSize; passing 0/NULL results in a failed decompression. All the decompression methods ask for compressedSize/originalSize, which is unknown but with a maximum value. – KKlouzal Jul 11 '16 at 21:29
  • Do you mean that you don't have the compressed size, decompressed size, or neither? – Tal Ben-Nun Jul 11 '16 at 22:14
  • I don't know the 'true' compressed size; It may be only 500 bytes, but is held inside a larger buffer of 1436 bytes. Passing 1436 as 'compressedSize' fails, where passing 500 will succeed. – KKlouzal Jul 11 '16 at 22:35
  • If the data is only partially given, not every compression algorithm will succeed (for example, if you break an LZ4 block you wouldn't be able to decompress it). However, if you know that entire blocks are given, why not use the stream-decoding functionality of LZ4 (`LZ4_createStreamDecode`)? – Tal Ben-Nun Jul 12 '16 at 08:54
  • See my edit for a code sample on streaming decompression with LZ4. – Tal Ben-Nun Jul 12 '16 at 09:03