Why does indy return unreadable data but chrome loads it fine?

Question

I'm trying to do a GET request on one of my game server's RESTAPI. When I load it in chrome it load with the correct json response but when I try to do it with indy it just gives me a bunch of unreadable text.

URL for the request: http://129.232.180.125:28019/deaths.json

Chrome Response:

Indy Reponse:

Y;9[ÿÍZË7¼ç3öìDQ%ÿJÃÚØ,Öw   òï§IµzvXÓö{Ab¿ÿöÏÝ»÷Ä!¾»ûûîýBó»»÷ßOwïï><¼îþ}§ På¢&È÷ÇÇÓË
gP¤@¹(¬Dv&©ÙEµ¸®D"`»@ëJ¥E\¦£??Þ?Ù@%¤3¨µ òrÿaCHZÝÀÕ
¯[%ö1cÓÔB)QV¡÷àõ=¢oÅt¥þóR§Ó§§Ï®Ô¶â*ëEu<wKÜôiýUØ^VµcÓ@vêâ»ÈR©hPædÛ=û±ô~¡;±8¢¢ÒtÇ/§û
Si¿·Õ
Zô1-W¥²¾KdP¡tÇXýc--[ÛàÔ-Go¼aI4]ñôãÏÇ¯Ï_& h¶@5Gá0Ú#!`u.éöæà¿i³ÇhFÜ]Q´µjÁ
d¼Ê%½|ÔRØ\¡Ókbê;Â$`¹^ìX»Ó`Ï¢Ñ¤ù»   i2S$Ýûöõ¯ÉÑ2    D(,öæÛý§ûÙ¯j\-AOÙ^&Gä£WK¥2\ÙW[%a»a\Ò' U½4ßß+¯>bV«ÏêUçq¡õÏIÜ
,1×çiJ¾¸¸^Ðh}_æéê{²Ú°pp¼²KÖ÷ÞNÄIß¼D÷W+K3ËûÔz:}ÚÞ5
SB^"ÕKªGè¸+¾ß^~$Õâ³6Tõ]2Õßr9¸%eàYYUrsJÏ`Êä³ë'£t8#®AB¨KÏ¹ëOAe9V ×Ís-£¬Óé¬ºùÜ\)µµ©±¢xÜj« ½ê(JRâÜ>\<M´@6Xd°ý~®i ÄCõBe;&©l³"¦}.uUè5(ÝE®V;BìðùôtúöðqÆÅ8~cÜ¡}{\h9Í©¡0²]Cèl¯s%qeÅô{Ø]zOöÞ|W-ÆöÎ%åª«¦@çÎÐ»[jÑ¶ÊÞÒÙq¤¸qZÛ/îÕ=KTÓ³ÿÒ=Go@Îì6*Z¯\W7¡ê!rM´'£¸ÇR»[G%ëp"?+²q?!ÔU{n,QâsðÚ9÷Þòì³H¡#f<Î.ÛÂÑ²Ät$u\9KoËGpE9b+ NsQÏ>ÉIiÒQs8ôòVÖMó.Þ÷ex ½½îx!RNÇZH?×b@ Ó$v¥_SfçF«Q®n`|à`«T®Ð¦©¸×5*<?qC´/ø©¿ÑJ¬ªï´´+ýÑHjiK( ;¤øÓA#mýHß¬5v ¡µR
´RI{ÊPAn,&æF ,&Õ/ônÓÚ¨oÄÀùYó}DUÿêà}C$e õ^LYæªYßÍX«Z2»ËeOæVqWvrW1ÚP«¿_kÆ-¢¢2Ar«ö©]x¢|4Ñ¿¯Y[¹">´¸(lÉÆÓÌå/BÙ!ìPGr£`¶Ê
[!ªóèsfÄt+.v.5_jÎê'Ô8Õ
Ö1ÒøMÜÑõâ[®ÇÄ)ãZYàÜ;Q!Ëa2Û^Zìçsá¹Ø^´QZbKnk9&ÄKºý¼ý®Úú11pÀ¦Eç°¿9r¾«ÿ®yýÆà"ø×
 (ù ?Ko¼Þ!Î¤¸ò¿^/ºo¼½åzãß*A9aãÔ°¥Ù8umÁàËuni(
    '-®®DÕêüý¸o¢zäÝ²é/lßþËÁi-ö:IhÙ8£i?|Û×RÐØï/ÐÀ,I
°Ù¥xXIRê0¹\x9séU=ªajä ^Æ£À±b¼$ÞP6<fF!µ¡Àßù!ûí7t¨¬XRÀJï\VãØsþð»b¨´7B` FýÀJÕ´F 
6Ë

èög»&·ã«¶ÛýAuR®ÝåL¬.YÌÄÿ£0æÀ%dÅ)²dhs´k¯.Ûg%ÅÂ.sâ¬Úé
zµ½ ;¸Z}»ÕUH«;ébÿ£j£%³S÷ú~X´@ýgý:¹½ 0w@ºØHâ³·,`!«G(]íä,Q¡NØm¢Î°,ÔY`Ùi;Ò¦×b¤×?BûMd¡Çdª <ï"QÕýWÏ®µ±× "6'3ý¹¦wþGSCt7Ùäk Ne¾ªÑ¿Ò(múS Ô$DifA{ª*ö¥(-gg¶a³~v(dê©=Om+á½üí±6®yf°1²ÐNi(y`B»ò¦¹ÃàuS!kµ¡¡i÷6¦   hÖ½;r"¢R»yÆ&OªoE·VU´¹)_Q9Ç¥2lÑÀâîUhñ~¼4óß4"\µnøæèî½5Ç00fúiPàýLØ~¯é§?_HÙ&Í÷;Ynnw¯8àoü¬¼^+2

My Code:

mmo1.Lines.Text := idhtp1.Get('http://129.232.180.125:28019/deaths.json');

I don't have access to that code. I'm using this documentation: http://playrust.io/manual/#!api.md — Adriaan, Jul 01 '18 at 18:50
Delphi version 10.2.3 and I have tried asking him but he says if it works fine in chrome then my code should work. — Adriaan, Jul 01 '18 at 18:52
Sorry if if it looks fake. I use cnwizard and it renames it like that automatically. I got used to having the name as ```idhtp``` the 1 is the component number. Here's a screenshot: https://i.imgur.com/yO4DRur.png — Adriaan, Jul 01 '18 at 18:58
the answer is easy, content is gzipped, so you must decompress... — whosrdaddy, Jul 01 '18 at 18:58
Confirmed, I see `Content-Encoding: gzip` in the headers. Chrome detects this automatically and handles it, but in Indy you must do this manually. — Jerry Dodge, Jul 01 '18 at 19:04
Wow. This is actually the first time I've heard of gzip. Thanks for all your info and help! — Adriaan, Jul 01 '18 at 19:11
@JerryDodge `TIdHTTP` can handle gzip compressed responses **IF** the `TIdHTTP.Compressor` property is assigned and enabled. DO NOT manually assign anything to the `TIdHTTP.Request.AcceptEncoding` property, let `TIdHTTP` manage that for you based on the capabilities of the assigned `Compressor` (or lack of one). Otherwise, if you manually set the `AcceptEncoding`, you must be prepared to manually detect and handle the response's encoding (gzip is just one of several possible encodings that modern web browsers support) — Remy Lebeau, Jul 01 '18 at 20:50
@Remy Indeed, assigning a component can be considered "manually", and I wasn't referring to changing any headers. I just meant there's extra work to do than just using `TIdHTTP` alone. — Jerry Dodge, Jul 01 '18 at 21:01
@JerryDodge My point is, the server **should not** be sending gzip compressed data unless the client **explicitly** asked for it via an `Accept-Encoding: gzip` request header (if the server does without that, then the server is buggy). So DO NOT specify that you accept an encoding that you are not actually prepared to handle. Let `TIdHTTP` handle the `Accept-Encoding` request header for you (which it does automatically when a `Compressor` is assigned). — Remy Lebeau, Jul 02 '18 at 18:06
Better do not use Indy components at all. Use native **THttpClient** from System.Net.HttpClient.pas . It works without any problems on Android 6 and higher - https (Indy does not ), do not need dlls and more stable. — alitrun, Jul 22 '18 at 10:19

score 13 · Accepted Answer · edited Jul 23 '18 at 22:06

If the server has Content-Encoding: gzip in the headers, so you must decompress the data. Here is a full & working example (the check for gzipped content is omitted):

program SO51126032;

{$APPTYPE CONSOLE}

{$R *.res}

uses
  IdHttp,
  IdZLib,
  Classes,
  System.SysUtils;

function GetServerData(Url : String) : String;
var
  Http : TIdHttp;
  Strm : TMemoryStream;
  OutStrm : TStringStream;
begin
  Http := TIdHttp.Create(nil);
  try
    Http.HandleRedirects := True;
    Http.Request.UserAgent := 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20100101 Firefox/12.0';
    Http.Request.AcceptEncoding := 'gzip';
    Strm := TMemoryStream.Create;
    try
      OutStrm := TStringStream.Create(TEncoding.UTF8);
      try
        Http.Get(URL, Strm);
        Strm.Position := 0;
        if TextIsSame(Http.Response.ContentEncoding, 'gzip') then
          DecompressStream(Strm, OutStrm)
        else
          OutStrm.CopyFrom(Strm, 0);
        end;
        Result := OutStrm.DataString;
      finally
        OutStrm.Free;
      end;
    finally
      Strm.Free;
    end;
  finally
    Http.Free;
  end;
end;

begin
  try
    Writeln(GetServerData('http://129.232.180.125:28019/deaths.json')); 
  except
    on E: Exception do
      Writeln(E.ClassName, ': ', E.Message);
  end;
  Readln;
end.

* RECOMMENDED WAY *

Indy can do the hard work for you if you assign a TIdCompressorZlib to TIdHTTP, it will automatically detect gzipped content and decompress on the fly:

uses
  IdHttp,
  IdCompressorZLib,
  System.SysUtils;

function GetServerData(const URL : String) : String;
var
  Http : TIdHttp;
begin
  Http := TIdHttp.Create(nil);
  try
    Http.HandleRedirects := True;
    Http.Request.UserAgent := 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20100101 Firefox/12.0';
    Http.Compressor := TIdCompressorZLib.Create(Http);
    Result := Http.Get(URL);
  finally
    Http.Free;
  end;
end;

I didn't downvote. But if you don't use `TIdHTTP.Compressor`, like in the 1st example, make sure you check the `TIdHTTP.Response.ContentEncoding` property for `'gzip'` before you call `DecompressStream()`. Just because you tell the server you support compression (and this example does not do that, FYI) doesn't mean the server will actually send a compressed response. — Remy Lebeau, Jul 23 '18 at 02:09
@RemyLebeau: I just responded with the OP's code, anyway I edited the answer to point future readers to the compressor example. — whosrdaddy, Jul 23 '18 at 06:58
I updated your example to look at the `ContentEncoding` property before manually decompressing. — Remy Lebeau, Jul 23 '18 at 22:07

Why does indy return unreadable data but chrome loads it fine?

1 Answers1

* RECOMMENDED WAY *