gzip decompression issue on OSX with TIdHTTP and TIdCompressorZLib

Question

I'm trying to fetch a page with gzip compression enabled, using TIdHTTP and TIdCompressorZLib. On Windows, the code works fine and the data is decompressed. But the exact same code on OSX is returning garbage data that looks like its still compressed. I can't really see where it's going wrong?

This is the code I'm testing with:

with TIdHTTP.Create(nil) do begin
  HandleRedirects := true;
  Compressor := TIdCompressorZLib.Create(nil);
  Request.AcceptEncoding := 'gzip, deflate';
  Data := Get('http://google.com.au');
  Compressor.Free;
  Free;
  WriteLn(Data);
end;

Data looks like the original compressed garbage on OSX, while it is plain decompressed HTML on Windows.

I'm using Delphi 10.1 Berlin Update 1 and OSX 10.11.

score 1 · Answer 1 · answered Oct 18 '16 at 19:49

You are manually setting the TIdHTTP.Request.AcceptEncoding property to tell the webserver that it is OK to send a compressed response even if TIdCompressorZLib is not actually ready to handle it. In your case, the TIdCompressorZLib.IsReady property is likely reporting False on OSX, but True on Windows.

In January 2016, Indy was updated to dynamically load the ZLib library on-demand the first time ZLib is used (SVN rev 5330). That change broke TIdCompressorZLib, which was later fixed in February 2016 (SVN rev 5343). I do not know if that fix is in Berlin or not. Try installing the latest SVN rev and see if the problem continues (instructions and download).

When using the TIdHTTP.Compressor property, DO NOT set the Request.AcceptEncoding property manually at all:

with TIdHTTP.Create(nil) do begin
  HandleRedirects := true;
  Compressor := TIdCompressorZLib.Create(nil);
  // Request.AcceptEncoding := 'gzip, deflate'; // <-- here
  Data := Get('http://google.com.au');
  Compressor.Free;
  Free;
  WriteLn(Data);
end;

Leave Request.AcceptEncoding blank, and let TIdHTTP update it internally if the assigned Compressor is actually ready to handle compressed responses.

BTW, you are leaking the TIdHTTP and TIdCompressorZLib objects if TIdHTTP.Get() raises an exception on failure. You should be using try/finally blocks:

with TIdHTTP.Create(nil) do
try
  HandleRedirects := true;
  Compressor := TIdCompressorZLib.Create(nil);
  try
    // Request.AcceptEncoding := 'gzip, deflate';
    Data := Get('http://google.com.au');
  finally
    Compressor.Free;
  end;
finally
  Free;
end;
WriteLn(Data);

Thanks. You are correct that the `IsReady` property was returning false. It constantly returns false after several http requests, as it appears the zlib library is not being loaded at all, which I presume is the bug introduced in Jan-2016 SVN rev 5330. I added the `IdZlibHeaders` unit to the Uses list, and called `IdZlibHeaders.Load` after creating the `TIdCompressorZLib` object, and now `IsReady` is returning true and gzip content is being decompressed — Sierra C, Oct 19 '16 at 04:13
Yes, the original bug was that `TIdCompressorZLib` expected `IdZLibHeaders.Load()` to have already been called, so `IsReady` simply returned the value from `IdZLibHeaders.Loaded()`. The fix was to make `IsReady` call `Load()` instead, in case it hadn't been called yet. — Remy Lebeau, Oct 19 '16 at 17:06

gzip decompression issue on OSX with TIdHTTP and TIdCompressorZLib

1 Answers1