1

I am trying to use Cryptsy api with this link: http://pubapi.cryptsy.com/api.php?method=orderdatav2

When I go there with my browser I get 236.2 KB response.

When I use php with curl/fopen/file_get_contents and produce a GET request I get 1.2M response with the same data! (which takes much longer to be downloaded).

I tried to copy the exact headers but still same results.

I suspect it is because the browser is using gzip and php is not?

Any other options? I am chasing my tail with this one.

The question is what causing this?

Yoni Hassin
  • 584
  • 1
  • 5
  • 17
  • I'm guessing HTML compression? –  May 30 '14 at 18:01
  • Most likely the compression is the difference, yes. Why don't you simply take a look at the raw data stream, then you know it? Just use a network sniffer or a debugging proxy. – arkascha May 30 '14 at 18:01
  • You have both downloads. Compare them. – Andy Lester May 30 '14 at 18:01
  • @AndyLester No, that won't say anything. The compressed data will be decompressed, the result _should_ be the same. That is the intention of a loss less compression. – arkascha May 30 '14 at 18:02
  • 1
    Visit the URL with a browser. Select all/copy. Paste into text editor. Save file. File size is 1.2MB. – James May 30 '14 at 18:08
  • 1
    @Yoni Hassin, if you happen to use Firefox + Firebug, you can see if the page served was compressed by looking at Net tab and it's headers (Content-encoding: gzip or similar) –  May 30 '14 at 18:15

2 Answers2

1

The reason is, your Browser sends the following Header Accept-Encoding: gzip, deflate.

The Server is checking for the existence of this header, and then compress the output with gzip.

Your Browser receives the content and decompress it, if needed.

CURL by default, doesn't. You can change this, with this Code:

curl_setopt($ch,CURLOPT_ENCODING , "gzip");

Also note, that this Option takes care about the complete process. You don't have to setup a custom Accept-Encoding Header or decompressing. You do not need to handle it.

More Information:

http://www.php.net/manual/en/book.curl.php

http://en.wikipedia.org/wiki/HTTP_compression

http://betterexplained.com/articles/how-to-optimize-your-site-with-gzip-compression/

Christian Gollhardt
  • 16,510
  • 17
  • 74
  • 111
0

Actually curl_setopt($ch,CURLOPT_ENCODING , "gzip"); didn't work for me.

The solution I found is written here php - Get compressed contents using cURL

$ch = curl_init("http://games2k.net/");
curl_setopt($ch,CURLOPT_ENCODING , "");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); 
$output = curl_exec($ch);
curl_close($ch);
echo $output;

You were right about the gzip :)

Community
  • 1
  • 1
Yoni Hassin
  • 584
  • 1
  • 5
  • 17
  • 1
    Strange, it should work with `gzip`. According to the manual: `If an empty string, "", is set, a header containing all supported encoding types is sent. ` – Christian Gollhardt May 31 '14 at 01:10