Can't get nginx to reply to ab with compressed content

Question

ab -n 1 -H 'Accept-Encoding: gzip, deflate' http://mywebsite.com/

This always returns the uncompressed content (I can see that both from its size and because I'm sniffing the traffic with ngrep).

However, if I request the content with Firefox, the result is compressed. Content-Encoding: gzip it says. I've tried to specify with ab all request headers Firefox specifies, as I see them sniffed, but still ab gets it uncompressed and Firefox gets it compressed.

I'm running a version of nginx which I have compiled myself but is identical to the Debian squeeze backports 1.2.1 version, except that it has the syslog patch. ApacheBench 2.3 revision 655654. Here is the nginx gzip configuration:

gzip                on;
gzip_min_length     1000;
gzip_types          text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
gzip_disable        "MSIE [1-6]\.(?!.*SV1)";
gzip_vary           on;

The document type is text/html (not listed above but always compressed by nginx; besides, I tried adding it, with no difference). It is dynamic, received from a uwsgi backend.

Sniffed request headers from ab:

GET [...] HTTP/1.0.
Accept-Encoding: gzip, deflate.
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8.
User-Agent: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:23.0) Gecko/20100101 Firefox/23.1.
Accept-Language: en-US,en;q=0.5.
Cookie: csrftoken=[...].
Authorization: Basic [....].
Connection: keep-alive.
Host: [...].

Sniffed request headers from Firefox:

GET [...] HTTP/1.1.
Host: [...].
User-Agent: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:23.0) Gecko/20100101 Firefox/23.0.
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8.
Accept-Language: en-US,en;q=0.5.
Accept-Encoding: gzip, deflate.
Cookie: csrftoken=[...].
Authorization: Basic [...].
Connection: keep-alive.

Hmmm, maybe it could be the protocol version.

No difference. Why should that make a difference anyway, given that Firefox says `Accept-Encoding: gzip, deflate`? — Antonis Christofides, Aug 12 '13 at 16:10
Please specify the versions of Nginx and 'ab' involved. Also, please include all the `gzip_*` directives from Nginx that you are using. — Mark Stosberg, Aug 12 '13 at 16:20
Also, please show the incoming HTTP request headers, as generated by Firefox, and as generated by 'ab'. — Mark Stosberg, Aug 12 '13 at 16:30

score 2 · Accepted Answer · edited Aug 13 '13 at 09:13

2

Indeed, the protocol version is the problem. As Mark Stosberg explains, whether HTTP 1.0 supports compression or not is not clear. What is clear is that, by default, nginx supports compression only for 1.1. This can be changed with a configuration option:

 gzip_http_version 1.0;

edited Aug 13 '13 at 09:13

VBart

8,309
3
25
26

answered Aug 13 '13 at 07:42

Antonis Christofides

2,598
2
23
35

score 1 · Answer 2 · answered Aug 12 '13 at 18:13

That appears that it could be an issue with HTTP 1.0 vs HTTP 1.1. Documentation about the differences include this:

HTTP/1.1 (unlike HTTP/1.0) carefully specifies the Accept-Encoding header, used by a client to indicate what content-codings it can handle, and which ones it prefers. One tricky issue is the need to support ``robot'' clients that are attempting to create mirrors of the origin server's resources; another problem is the need to interoperate with HTTP/1.0 implementations, for which Accept-Encoding was poorly specified.

Here's an idea for a cross-check Take Firefox and ab out of the equation and send raw HTTP requests using a purpose-built tool for that, like GET from the libww-perl distribution. Here's an example syntax to get started:

GET -sSe -H 'Connection: keep-alive' -H 'Accept-Encoding: gzip, default'  'http://example.com' | head -40

You can see how to add more headers to make it match your request from there.

Can't get nginx to reply to ab with compressed content

2 Answers2