Unexpected http/net response content length in Golang

Question

I am making a HTTP Get request using the code below. I proxy it through Fiddler to analyse the request. The request seems to be being made correctly, with the expected response. However, the resp.ContentLength property of the response in GO is -1 every time despite the fact that the response in Fiddler shows a positive integer value of Content-Length: 1830.

Why is the ContentLength not being picked up in GO?

Go Code

package main

import "net/http"
import "os"
import "fmt"

func main() {
    os.Setenv("HTTP_PROXY", "http://127.0.0.1:8888") // For Fiddler debugging
    resp,err := http.Get("http://www.google.com/robots.txt")
    if (err == nil) {
        if (resp.StatusCode == 200) {
            fmt.Println(resp.ContentLength) // Prints -1
        }
    }
}

Resulting Request in Fiddler

GET /robots.txt HTTP/1.1
Host: www.google.com
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip

Response Received in Fiddler

HTTP/1.1 200 OK
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Type: text/plain
Content-Length: 6907
Date: Mon, 05 Mar 2018 13:53:00 GMT
Expires: Mon, 05 Mar 2018 13:53:00 GMT
Cache-Control: private, max-age=0
Last-Modified: Thu, 01 Mar 2018 18:00:00 GMT
X-Content-Type-Options: nosniff
Server: sffe
X-XSS-Protection: 1; mode=block

check the answer down this question. it worked for me. https://stackoverflow.com/a/49487632/19632867 — Yuvalk, Jul 27 '22 at 12:41

score 10 · Accepted Answer · answered Mar 05 '18 at 14:20

10

The value -1 indicates that the length is unknown. (in most cases this indicates chunked encoding).

answered Mar 05 '18 at 14:20

Roman Kiselenko

43,210
9
91
103

That would make sense! Although I was under the impression that the server should return a 'chunked' header in the response if that is the case. – Jordan Mar 05 '18 at 14:20
1

You can find more in the source https://golang.org/src/net/http/response.go – Roman Kiselenko Mar 05 '18 at 14:27

score 1 · Answer 2 · answered Jun 21 '20 at 10:08

Just want to share my research.

There's a DisableCompression option in the http.Transport struct, which default to false (http.Get use this default option). With that option, go's internal package will forcing gzip compression by adding the Accept-Encoding: gzip header in the request. After receiving the response, go's internal package will do the decompression job for us, so that we don't have to deal with the gzip chunk. They also delete the original Content-Length header because it's not correct anymore.

All of that "magic under the hood" behaviors give us what we see here, missing the Content-Length header.

Ref: https://github.com/golang/go/blob/master/src/net/http/transport.go#L182

I set the `DisableCompression` to `true`, still getting the content-length as 0. — Syed Qasim Rizvi, Oct 05 '20 at 19:14

Alan Gus · Answer 3 · 2022-08-09T06:53:41.580

0

The content length change in func readTransfer https://github.com/golang/go/blob/master/src/net/http/transfer.go

edited Aug 09 '22 at 06:53

answered Aug 09 '22 at 06:49

Alan Gus

1
2

As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Aug 13 '22 at 03:01

Unexpected http/net response content length in Golang

3 Answers3