Uncompressing a LZ4 blob with Perl

Question

I have a table in a SQLite db that stores blobs compressed with LZ4 algorithm. I am trying to use decompress/uncompress functions from Compress::LZ4, but not getting any success with it.

The sample SQLite db can be downloaded from here.

Here is how I am connecting to the SQLite db and getting the blob:

use DBI; 
use Data::Dump; 
use MIME::Base64; 
use Compress::LZ4;    

my $dbh = DBI->connect("dbi:SQLite:dbname=$ARGV[0]","","");   
$sth = $dbh->prepare("select blob_data from blob_parts where data_fk = 6");
$sth->execute();   
$result = $sth->fetch;   
$blob = $result->[0];

dd $blob;
dd (decompress($blob));

$sth->finish();  
$dbh->disconnect;

For the particular blob that I am selecting in this sample code (data_fk=6), dd outputs the following:

"LZ4\1>\1\0\0\xF7\xD6df\xF1mBXML\1\xA1\aVersion\xA1\4Type\xA1\2Id\xA1\3Ref\xA1\4Size\xA1\3use\xA1\4expr\xA1\5value\xA1\4data\xA1/Serialization\xA1\aPoints3\xA1\tuser_ E\0\xF0\16\bvertices\xA1\6double\xA1\bhas_attr\xA1\16\n\0\xC7object_ids\xA1\n\f\0\xF1M\4item\xA1\tis_active\xA0~B\20\n\22\6\4\x8C\1\0\0\0\6\2\xAA\24\6\0\xA4\x82\x88\2\x80\x82\x82\xA6B\26\6\b\x80\1B\30 \6\b\x88\2\3B\32\1\x93\6\0\0\0`\xACu\xCF\xBF\0\0\0\0\xCC\xF8\xC2?\0\0\0\0\0\x004\@\0\0\0 \xAA\xEF\xA9\20\x001h\xC5\xB1\b\0\xD0\0\0\$\@\1B\34\x85B\36\x87B C\0\xF0\aB\"\6\0\x88\3B\$\x85\1B\"\6\0\x88\3B\ $\x85\1\1\1"

But the decompress/uncompress functions just return undef. The uncompressed data should be something like (The following output is generated by a XML converter):

<?xml version="1.0" encoding="utf-8"?>
<MultiStreamDocument>
<!-- Stream 1 -->
<?xml version="1.0" encoding="utf-8"?>
<data xmlns="" Id="1" Type="Points3" Version="1 2 0 1 1">
    <user_data Size="0"></user_data>
    <vertices Size="2">
        <double>-0.24577860534191132</double>
        <double>0.14821767807006836</double>
        <double>20</double>
        <double>0.050656620413064957</double>
        <double>0.069418430328369141</double>
        <double>10</double>
    </vertices>
    <has_attr>false</has_attr>
    <has_object_ids>true</has_object_ids>
    <object_ids Size="2">
        <item Version="3">
            <is_active>false</is_active>
        </item>
        <item Version="3">
            <is_active>false</is_active>
        </item>
    </object_ids>
</data><!-- Stream size: 126 bytes -->
</MultiStreamDocument>

What is the correct way to get uncompressed blob data from this SQLite database?

What method are you using to compress the data? It looks to be that the compression algorithms are incompatible. — bolav, Dec 25 '15 at 11:30
I do not know about the specific method, the only information I have is that it is LZ4 compressed. — user2006190, Dec 25 '15 at 12:48
Look at this [mail](http://mx.gw.com/pipermail/file/2014/001299.html) regarding magic. Your example looks like it has some magic (LZ4), so it doesn't look like a standard compression. — bolav, Dec 25 '15 at 19:24
Umm, Hey! Sooooo, did you have any luck with this or not? Cuz I'm doing the exact same thing. Of course a java library managed to decompress the data with 12 bytes offset, but since the uncompressed data is a BXML, you gotta do it in C#. Anyways, I'd love it if you tell me whether you got this working or not. — Mostafa Zeinali, May 19 '19 at 08:19

user75857 · Accepted Answer · 2015-12-25T21:57:19.887

You data looks like it is LZ4-compressed and prefixed with the four bytes "LZ4\1" presumably as a format indicator

The next four bytes ">\1\0\0" are a little-endian original-size field which evaluates to 318 bytes, which is reasonable. The decompress library function expects this field

So in theory, you should be able to write

$blob = substr($blob(4);
dd decompress($blob);

and get the correct result. However this also results in a value of undef for me, which suggests that the data is corrupted somehow

What is certain is that most of the data has ended up uncompressed. The two bytes following the length field are "\xF7\xD6", which indicates that the data following that is 229 bytes of literal data (the upper nybl of the first byte - 0xF - plus the second byte - 0xD6 - is 0xE5 or 229). So this part of the data

"df\xF1mBXML\1\xA1\aVersion\xA1\4Type\xA1\2Id\xA1\3Ref\xA1\4Size\xA1\3use\xA1\4expr\xA1\5value\xA1\4data\xA1/http://www.slb.com/Petrel/2011/03/Serialization\xA1\aPoints3\xA1\tuser_E\0\xF0\16\bvertices\xA1\6double\xA1\bhas_attr\xA1\16\n\0\xC7object_ids\xA1\n\f\0\xF1M\4item\xA1\tis_active\xA0~B\20\n\22\6\4\x8C\1\0\0\0\6\2\xAA\24\6\0\xA4\x82\x88\2\x80\x82\x82\xA6B\26\6\b\x80\1"

is literal, as could be guessed by the amount of readable text it contains

The following two bytes, "B\30" should indicate an offset within the translated buffer from which data should be copied. Unfortunately this evaluates to 6210, whereas, as we have seen, the buffer is only 229 bytes long so far. This is presumably where the data causes the decompress function to balk and return undef

That's the best I can make of your data. I hope it helps

Thanks for pointing in this direction. I spent the day trying to understand the structure of LZ4 compressed data and now I see why it isn't working. — user2006190, Dec 26 '15 at 16:46

Uncompressing a LZ4 blob with Perl

1 Answers1