Writing a compressed buffer to a gzip compatible file

Question

I would like to compress a bunch of data in a buffer and write to a file such that it is gzip compatible. The reason for doing this is that I have multiple threads that can be compressing their own data in parallel and require a lock only when writing to the common output file.

I have some dummy code below based on the zlib.h docs for writing a gz compatible, but I get a gzip: test.gz: unexpected end of file when I try to decompress the output. Can anyone tell me what might be going wrong ?

Thank you

#include <cassert>
#include <fstream>
#include <string.h>
#include <zlib.h>


int main()
{

    char compress_in[50] = "I waaaaaaaant tooooooo beeeee compressssssed";
    char compress_out[100];

    z_stream bufstream;
    bufstream.zalloc = Z_NULL;
    bufstream.zfree = Z_NULL;
    bufstream.opaque = Z_NULL;

    bufstream.avail_in = ( uInt )strlen(compress_in) + 1;
    bufstream.next_in = ( Bytef * ) compress_in;
    bufstream.avail_out = ( uInt )sizeof( compress_out );
    bufstream.next_out = ( Bytef * ) compress_out;

    int res = deflateInit2( &bufstream, Z_DEFAULT_COMPRESSION, Z_DEFLATED, 15 + 16, 8, Z_DEFAULT_STRATEGY );
    assert ( res == Z_OK );

    res = deflate( &bufstream, Z_FINISH );
    assert( res == Z_STREAM_END );

    deflateEnd( &bufstream );

    std::ofstream outfile( "test.gz", std::ios::binary | std::ios::out );
    outfile.write( compress_out, strlen( compress_out ) + 1 );
    outfile.close();

    return 0;
}

Your code is C++. Please don't tag other vastly different languages. — Some programmer dude, Oct 19 '22 at 15:25
can you be 100% certain that `compress_out` contains no embedded null characters and that `strlen( compress_out )` will return the correct value? I'm not sure that you can. — user4581301, Oct 19 '22 at 15:27
You can't use `strlen` on binary data. You need to use the sizes returned by zlib instead — Alan Birtles, Oct 19 '22 at 15:31
I can't find a good documentation page for `z_stream` to confirm which of it's members you want to use to determine the amount of output data. `next_out` and `total_out` sound like good candidates, but I'm out of time and cannot confirm. — user4581301, Oct 19 '22 at 15:45

score 2 · Accepted Answer · answered Oct 19 '22 at 19:45

The length of the compressed data written to the output buffer is the space you provided for the output buffer minus the space remaining in the output buffer. So, sizeof( compress_out ) - bufstream.avail_out. Do:

outfile.write( compress_out, sizeof( compress_out ) - bufstream.avail_out );

Writing a compressed buffer to a gzip compatible file

1 Answers1