Reading gzipped HTTP response in Go

Question

I am trying to read a gzipped HTTP response with Go! but I always get the following error message :

panic: gzip: invalid header
[...] stack trace [...]

If I run "curl -H "Accept-Encoding: gzip" http://foo.com/ | gunzip -" I get the response correctly gunzipped. I also double checked with ngrep and the pair Accept-Encoding/Content-Encoding is correctly sent/returned.

If I create a file with some dummy content and gzip it, I can read it from my Go! program.

The program I used for testing:

package main

import (
    "io"
    //"os"
    "fmt"
    "compress/gzip"
    "net/http"
)

func main() {
    /* This works fine
    f, _ := os.Open("/tmp/test.gz")
    defer f.Close()
    reader, err := gzip.NewReader(f)
    */

    // This does not :/
    resp, _ := http.Get("http://foo.com/")
    defer resp.Body.Close()
    reader, err := gzip.NewReader(resp.Body)

    if err != nil { panic(err) }

    buff := make([]byte, 1024)
    for {
        n, err := reader.Read(buff)

        if err != nil && err != io.EOF {
            panic(err)
        }

        if n == 0 {
            break
        }
    }

    s := fmt.Sprintf("%s", buff)
    fmt.Println(s)
}

Have I overlooked something ?

Stephen Weinberg · Answer 1 · 2015-01-20T19:04:43.913

71

EDIT: The following is an example of manually handling compression. If you don't set the header, the default Transport will do it for you and then decompress while you read the response.Body.

client := new(http.Client)

request, err := http.NewRequest("GET", "http://stackoverflow.com", nil)
request.Header.Add("Accept-Encoding", "gzip")

response, err := client.Do(request)
defer response.Body.Close()

// Check that the server actually sent compressed data
var reader io.ReadCloser
switch response.Header.Get("Content-Encoding") {
case "gzip":
    reader, err = gzip.NewReader(response.Body)
    defer reader.Close()
default:
    reader = response.Body
}

io.Copy(os.Stdout, reader) // print html to standard out

Error handling removed for brevity. I kept the defers.

edited Jan 20 '15 at 19:04

answered Oct 30 '12 at 00:58

Stephen Weinberg

51,320
14
134
113

http.Get seems to send Content-Encoding automatically (unless I am mistaken). I tried with the following code : http://play.golang.org/p/haoPEZV6_H and running "sudo ngrep -W byline -d en1 port 80" returns me this : http://pastebin.com/PqhCX8gQ . We clearly see and Accept-Encoding header. However if you run http://play.golang.org/p/haoPEZV6_H do you get a runtime error ? Maybe I misuse the net/http API after all. – Jérôme R Oct 30 '12 at 07:52
1

Great example thanks! I had a play around with it - here is the complete script http://play.golang.org/p/Dmf06rhhcs – Nick Craig-Wood Oct 30 '12 at 08:14
The first argument to `http.NewRequest` should be fully capitalized for broadest compatibility. The example here didn't work for me with NginX. – jelder Jan 19 '15 at 16:46
It should be noted that at times, some APIs may send compresses responses even if not asked for by setting "Accept-Encoding: gzip". In such a case, Go Transport will not automatically decompress it, since it did not explicitly ask for it. More here: https://github.com/golang/go/issues/13298 – Abdul Wasae May 18 '17 at 11:14

score 37 · Answer 2 · edited Nov 02 '19 at 07:49

37

net/http#Transport handles gzip compressed responses. You don't have to do anything special.

Take a look at the DisableCompression option, here.

edited Nov 02 '19 at 07:49

Inanc Gumus

25,195
9
85
101

answered Oct 29 '12 at 22:08

simonmenke

2,819
19
28

I found nothing in the documentation about that. Do you have any pointers ? Thanks :) – Jérôme R Oct 30 '12 at 08:00
Hmmm... Strange... `net/http.Transport` has a field called `DisableCompression`. And I had to turn it off a while ago to prevent it from decompressing a gzip file send back by sourceforge. – simonmenke Oct 30 '12 at 08:52
2

I am up-voting. The standard client Transport handles gzip for you. This includes automatically decompressing the data as you read from response.Body. The OP's problem was that he was trying to decompress a second time. – Stephen Weinberg Oct 30 '12 at 15:11
1

If you want to manually decompress the data, you need to manually ask for data gziped. See my answer for an example. – Stephen Weinberg Oct 30 '12 at 15:18
3

It doesn't appear to handle `deflate`, though – Arnaud Le Blanc Dec 05 '17 at 16:39
This answer is way too short to be helpful for people with less background on this topic, I would suggest the answer to at least include a reference to relevant piece of comment, https://golang.org/src/net/http/transport.go#L181 – Daniel Dror Aug 27 '19 at 11:28

score 22 · Answer 3 · answered Aug 15 '16 at 11:47

22

According to net/http docs (line 110) if you manually set the Accept-Encoding request header, than gzipped response will not automatically decompressed by the http.Transport. Otherwise, behavior is controlled by the Transport's DisableCompression boolean

answered Aug 15 '16 at 11:47

Requilence

221
2
3

In the meanwhile, line numbers changed, moved to https://golang.org/src/net/http/transport.go#L181 – Daniel Dror Aug 27 '19 at 11:25

score 0 · Answer 4 · answered Dec 07 '21 at 08:51

The following is an example of manually handling compression.


import "compress/gzip"

func ReadAll(r io.Reader) ([]byte, error) {
    reader, err := gzip.NewReader(r)
    if err != nil {
        return nil, err
    }
    defer reader.Close()
    buff, err := ioutil.ReadAll(reader)
    return buff, err
}

Reading gzipped HTTP response in Go

4 Answers4

Linked

Related