2

I need help on decompressing chunk data from a stream api. I am connecting to Gnip stream api which returns json output using gzip compression. When I try to decompress the output data, it throws me the following error "Zlib::DataError: incorrect header check".

It might be very close to this issue - http://groups.google.com/group/nodejs/browse_thread/thread/32b9ee7f691a68d9

Here I attached my code snippets for your reference:

require 'rubygems'
require 'curl'
require 'stringio'
require 'zlib'
url = "https://stream.gnip.com:443/accounts/SomeGroup/publishers/twitter/streams/track/Prod.json"
crl = Curl::Easy.new(url)
crl.headers={"Authorization"=>"Basic dmVlcmFzd5kYXJhdmVsLRoaX1Z25hbmFzd5kYhbU4ZXJeC5b26GpbFnW0MzIy", "Accept-Encoding" => "deflate, gzip"}
zstream = Zlib::Inflate.new
crl.on_body { |data| zstream.inflate(data);}
crl.http_get

The above code always returns "Zlib::DataError: incorrect header check". I know the gnip returns the data chunk by chunk so the required gzip'ed output will not be in the first chunk. So how can I collect all required chunk of gzip'ed outputs and decompress them to get required single json output.

Thanks in Advance. Veeraa.

Veeraa
  • 29
  • 2

1 Answers1

3

By default zlib is looking for a zlib header, not a gzip header. So a gzip header would cause an incorrect header check. I don't know what the Ruby interface to zlib is like, but you should see if you can specify the type of stream to inflate. zlib's inflate supports zlib, gzip, and raw deflate streams. It also optionally provides an auto-detect of zlib and gzip streams.

Mark Adler
  • 101,978
  • 13
  • 118
  • 158
  • It looks like the `window_bits` parameter to `Zlib::Inflate.new` is passed directly to zlib, so it has the exact same meaning as `windowBits` in zlib. So I'd try a value of 31 (maximum window size and gzip) or 47 (maximum window size with automatic detection). – L2G Mar 07 '14 at 00:54