5

I am trying to decompress a content of unknown size using python-lz4 using the following code

with open("compressed.msgpk", "rb") as f:
    content = f.read()
    if content[0] == 1:
        uncompressed = lz4.block.decompress(content[1:])

but it always fails with

LZ4BlockError: Decompression failed: corrupt input or insufficient space in destination buffer. Error code: 58

I even tried specifying different/bigger sizes as shown here https://python-lz4.readthedocs.io/en/stable/lz4.block.html but nothing worked.

And if it help the content I am trying to decompress is compressed using lz4net c# library using method LZ4Codec.WrapHC(content) https://github.com/MiloszKrajewski/lz4net/blob/201ed085fed299523616bfd08776694cb61ae6b3/src/LZ4/LZ4Codec.cs#L562

Waqas
  • 6,812
  • 2
  • 33
  • 50
  • It doesn't seem likely to be a problem, but the 'unknown length' bit has me wondering if the file is larger than the amount of memory your python interpreter has available. Any chance that could be the case? – Bill Horvath Nov 08 '21 at 16:29
  • i don't think so, the maximum file size i noticed is like 20mb – Waqas Nov 09 '21 at 00:10
  • Would turning buffering off in the `open` statement make a difference? i.e. `with open(file="compressed.msgpk", mode="rb", buffering=0) as f:` Buffering is on by default, and that *might* explain the error suggesting the data is corrupted. – Bill Horvath Nov 09 '21 at 18:59
  • unfortunately, it didn't work – Waqas Nov 09 '21 at 23:29
  • Would you be able to share the compressed.msgpk file? – Dario Petrillo Nov 10 '21 at 02:18
  • When you say you tried specifying different/bigger sizes but nothing worked, can you be more specific? What size range did you try, and what was the increment? Is it possible that you introduced another error in implementing this approach? – Matt L. Nov 12 '21 at 00:07
  • "And if it help the content I am trying to decompress is compressed using lz4net c# library using method LZ4Codec.Unwrap(content, 1)" `LZ4Codec.Unwrap` doesn't compress - it is decompress function – Alexander Ushakov Nov 14 '21 at 10:02
  • @AlexanderUshakov it is using `WrapHC` to compress, I have made the changes to the question- thanks for pointing it out – Waqas Nov 15 '21 at 02:32
  • did you manage to solve this? same issue for me – Samuel Aug 04 '22 at 15:40
  • nope, didn't manage to solve this. – Waqas Apr 13 '23 at 01:55

1 Answers1

1

The unwrap method decodes a wrapped block, but the lz4.block.decompress method appears not to take wrapping into account.

I'm not 100% familiar with the python and c# libraries you are using but I wonder (from the docs) if the lz4.frame.decompress might be the method you are looking for?

Haqa
  • 476
  • 3
  • 14