Compressing/decompressing gzip into memory with boost

Question

I'll be brief: I have this piece of code:

QByteArray MyNBT::decompressData(QByteArray data)
{
    filtering_streambuf<input> in;

    std::string _data = data.data();

    in.push( gzip_decompressor() );
    in.push( boost::iostreams::back_inserter(_data) );
    //in.push( std::back_inserter(_data) );

    std::stringstream _sstream;
    boost::iostreams::copy(in, _sstream);

    QByteArray out = _sstream.rdbuf()->str().c_str();

    return out;
}

And it gives an error at this line(s):

in.push( boost::iostreams::back_inserter(_data) );
//in.push( std::back_inserter(_data) );

The error is:

/usr/include/boost/iostreams/chain.hpp:244: error: invalid application of 'sizeof' to incomplete type 'boost::STATIC_ASSERTION_FAILURE<false>'
     BOOST_STATIC_ASSERT((is_convertible<category, Mode>::value));
     ^

The compiler throws this error once in std::back_inserter(_data) and twice with boost's one.

Thanks in advance.

sehe · Accepted Answer · 2023-06-05T10:36:24.577

4

What does a back_inserter do?

Indeed. It inserts elements at the back of a container.

What you seem to be after, though, is a "front_reader" or container_source.

Well, I don't have Qt, but I had luck using array_source to adapt your input (note how it works equally well with std::string, std::vector, std::array or even just const char [] input):

Live On Coliru

#include <fstream>
#include <iostream>

#include <boost/iostreams/filtering_stream.hpp>
#include <boost/iostreams/filter/gzip.hpp>
#include <boost/iostreams/copy.hpp>
#include <sstream>

int main()  {
    using namespace boost::iostreams;
    filtering_streambuf<input> in;

#if 0
    std::string _data { 
#else
    std::vector<char> _data { 
#endif
        char(0x1f), char(0x8b), char(0x08), char(0x00), char(0xca), char(0xb5),
        char(0x07), char(0x53), char(0x00), char(0x03), char(0xcb), char(0x48),
        char(0xcd), char(0xc9), char(0xc9), char(0x57), char(0x28), char(0xcf),
        char(0x2f), char(0xca), char(0x49), char(0xe1), char(0x02), char(0x00),
        char(0x2d), char(0x3b), char(0x08), char(0xaf), char(0x0c), char(0x00),
        char(0x00), char(0x00)
    };

    in.push( gzip_decompressor() );
    in.push( boost::iostreams::array_source(_data.data(), _data.size()) );

    std::stringstream _sstream;
    boost::iostreams::copy(in, _sstream);

    std::cout << _sstream.rdbuf();
}

The output of the program is, of course, hello world

edited Jun 05 '23 at 10:36

answered Feb 21 '14 at 21:09

sehe

374,641
47
450
633

It compiles now, but it throws an exception at runtime `gzip error`. The gzips I used are OK. I've tried your code, some gzips made by me (and tested with other examples that worked)... – azteca1998 Feb 22 '14 at 09:54
I'm assuming you're referring to your code now, not mine (i tested it). I can only guess that the zip data is corrupted. Maybe you fail to use binary streams or running into terminating NUL characters somewhere along the road – sehe Feb 22 '14 at 09:57
My code read all the binary file into a string, pass it between some functions (without changing it) and then if I marked it as compressed, the function above decompress it. Quite simple. – azteca1998 Feb 22 '14 at 10:06
Time to check that the data is still in order when in the string (the exception tells you it isn't!). Perhaps write it to another binary file and uncompress that with external tools. Proof helps – sehe Feb 22 '14 at 10:10
I already tested that: Doesn't work. I reversed string with `std::reverse` on `` and no luck. I'm going to dump the compressed data and compare it with the original. – azteca1998 Feb 22 '14 at 10:14
You what? I don't see how reversing would be a realistic failure mode :) anyways, you're bound to find it now – sehe Feb 22 '14 at 10:16
It could be possible that it got reversed or something. In an example I found the `boost::iostreams::back_inserter`... Nobody knows :P Anyway I've found an error on my code. I don't know why this `std::string _data = data.data();` only copies a few bytes. I've used a `std::vector` and now I have all the bytes and my code doesn't throw an exception, but the decompressed data is only 1 byte :( – azteca1998 Feb 22 '14 at 10:33
2

@azteca1998 I suggest you try to get away from the "programming by coincidence" mindset ("Nobody knows" - erm. People do know. And the samples aren't wrong). Now "only copies a few bytes" is ***exactly*** what I predicted in my very first comment: you corrupted the input data by voluntarily choking on a NUL character. You just can't treat binary data as a C string. Not to mention the inefficiency. – sehe Feb 22 '14 at 20:13

score 0 · Answer 2 · answered Feb 22 '14 at 10:54

Fixed it using @sehe 's answer.

Here's the resulting code: (hope it'll be useful for people with the same errors)

QByteArray MyNBT::decompressData(QByteArray data)
{
    filtering_streambuf<input> in;

    std::vector<char> _data;
    foreach( char ch, data )
        _data.push_back(ch);

    in.push( gzip_decompressor() );
    in.push( boost::iostreams::array_source(_data.data(), _data.size()) );

    QByteArray out;
    while ( in.sgetc() != -1 )
        out += (unsigned char)in.sbumpc();

    std::cout << "out size! :: " << in.in_avail() << std::endl;
    //std::cout << "out size! :: " << _sstream.str().size() << std::endl;
    std::cout << "out size! :: " << out.size() << std::endl;

    return out;
}

Have a look at `std::copy_n` which is gonna be much more efficient. Likely, you can even just `std::vector const _data(data.begin(), data.end());` (note the const), you should be able to use `std::vector const _data(data.data(), data.size());` and finally, why don't you just use `array_source(data.data(), data.size())` directly without copying? — sehe, Feb 22 '14 at 21:12

Tim Price · Answer 3 · 2015-11-04T13:42:18.417

I also use a simple character array, and for efficiency's sake did not want to have to create a vector and copy the buffer contents. So,here's my take on a simple in-ram compress/decompress utility class that decompresses or compresses directly from the char array read from the wire into another char array:

/*
 * GzipUtil.h
 *
 *  Created on: Nov 3, 2015
 *      Author: tprice
 */

#ifndef GZIPUTIL_H_
#define GZIPUTIL_H_

#include <unistd.h>
#include <stdint.h>
#include <vector>
#include <stdlib.h>

#include <boost/iostreams/filtering_stream.hpp>
#include <boost/iostreams/filter/gzip.hpp>

namespace BIO = boost::iostreams;

class GzipUtil {
public:
GzipUtil() {}
virtual ~GzipUtil() {}

static ssize_t compress(char *src, char *dest, ssize_t len)
{
    ssize_t retval = 0;

    BIO::filtering_streambuf<BIO::input> in;
    in.push( BIO::gzip_compressor() );
    in.push( BIO::array_source(src, len) );
    for(retval = 0; in.sgetc() != -1; retval++ )
    {
        dest[retval] = (char) in.sbumpc();
    }

    return retval;
}

static ssize_t decompress(char *src, ssize_t srclen, std::vector<char*> & dest, ssize_t destChunkLen)
{
    ssize_t retval = 0;
    char*   chunkPtr = NULL;
    ssize_t chunkUsed = 0;

    BIO::filtering_streambuf<BIO::input> in;
    in.push( BIO::gzip_decompressor() );
    in.push( BIO::array_source(src, srclen) );
    for(retval = 0; in.sgetc() != -1; retval++ )
    {
        if((chunkPtr == NULL) || (chunkUsed >= destChunkLen))
        {
            chunkPtr = (char*) malloc(destChunkLen);
            dest.push_back(chunkPtr);
            chunkUsed = 0;
        }
        chunkPtr[chunkUsed++] = (char) in.sbumpc();
    }

    return retval;
}

};


#endif /* GZIPUTIL_H_ */

Compressing/decompressing gzip into memory with boost

3 Answers3