When using go http client Do method, can the httpResponse and the error be not nil at the same time?

Question

As visible in the official documentation as well as almost everywhere else online, the common pattern for handling http client errors is the following:

req, err := http.NewRequest("GET", "http://example.com", nil)
req.Header.Add("If-None-Match", `W/"wyzzy"`)
resp, err := client.Do(req)
if err != nil {
    // handle error
}
defer resp.Body.Close()

Reading the docs about the methods on the http client, I'm not able to understand if it is possible to receive both resp and err not nil, it seems like it's possible if we consider what is written in the Do method documentation:

The request Body, if non-nil, will be closed by the underlying Transport, even on errors.

On error, any Response can be ignored. A non-nil Response with a non-nil error only occurs when CheckRedirect fails, and even then the returned Response.Body is already closed.

Therefore my questions are the following:

Is it possible to receive non-nil resp and err at the same time?
If it is possible, and in my program I end up receiving a lot of errors, and I return or panic in the //handle error piece of the code shown above, thus never reaching the defer right after, is it possible that on the long run I end up having too many open bodies that the transport hasn't been able to close?

Context Addendum :

The problem I encounter is that, after a while, the program crashes with the following big print of goroutines on the stdErr, and following the trace I end up to a piece of code that seems normal, unless something bad happens if an error is encountered and there's a responseBody upon which no defer close will be made, but since the official docs say not to worry about it, the problem must be elsewhere...

Stderr logs
  Display timestamps
 net/http.(*persistConn).readLoop(0xc4244e0240)
    /usr/local/go/src/net/http/transport.go:1474 +0x196
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1117 +0xa35

goroutine 194055 [select, 2 minutes]:
net/http.(*persistConn).readLoop(0xc42421ec60)
    /usr/local/go/src/net/http/transport.go:1599 +0x9ec
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1117 +0xa35

goroutine 215944 [IO wait]:
net.runtime_pollWait(0x7f63736f0628, 0x72, 0x21b)
    /usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc420963568, 0x72, 0xa85c40, 0xa81350)
    /usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc420963568, 0xc422289000, 0x1000)
    /usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc420963500, 0xc422289000, 0x1000, 0x1000, 0x0, 0xa85c40, 0xa81350)
    /usr/local/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc42025bbb8, 0xc422289000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:181 +0x70
net/http.(*persistConn).Read(0xc4217af8c0, 0xc422289000, 0x1000, 0x1000, 0xc420b33400, 0x8b61a3, 0xc423dd6b80)
    /usr/local/go/src/net/http/transport.go:1316 +0x14b
bufio.(*Reader).fill(0xc421af0060)
    /usr/local/go/src/bufio/bufio.go:97 +0x117
bufio.(*Reader).Peek(0xc421af0060, 0x1, 0xc421af01e0, 0xc420536c80, 0xc420b17e00, 0x60, 0xc420536c60)
    /usr/local/go/src/bufio/bufio.go:129 +0x67
net/http.(*persistConn).readLoop(0xc4217af8c0)
    /usr/local/go/src/net/http/transport.go:1474 +0x196
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1117 +0xa35

goroutine 199855 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc421db1320)
    /usr/local/go/src/net/http/transport.go:1704 +0x43a
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1118 +0xa5a

goroutine 198707 [select, 1 minutes]:
net/http.(*persistConn).readLoop(0xc4236ea000)
    /usr/local/go/src/net/http/transport.go:1599 +0x9ec
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1117 +0xa35

goroutine 192577 [select, 2 minutes]:
net/http.(*persistConn).readLoop(0xc424a8fc20)
    /usr/local/go/src/net/http/transport.go:1599 +0x9ec
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1117 +0xa35

goroutine 213681 [select]:
net/http.(*persistConn).writeLoop(0xc42455e6c0)
    /usr/local/go/src/net/http/transport.go:1704 +0x43a
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1118 +0xa5a

goroutine 190222 [select, 2 minutes]:
net/http.(*persistConn).writeLoop(0xc4238617a0)
    /usr/local/go/src/net/http/transport.go:1704 +0x43a
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1118 +0xa5a

goroutine 208970 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc4234530e0)
    /usr/local/go/src/net/http/transport.go:1704 +0x43a
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1118 +0xa5a

goroutine 219699 [select]:
net/http.(*persistConn).writeLoop(0xc42155cb40)
    /usr/local/go/src/net/http/transport.go:1704 +0x43a
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1118 +0xa5a

goroutine 221664 [select]:
net/http.setRequestCancel.func3(0x0, 0xc4231af680, 0xc423657000, 0xc4214aeecc, 0xc422f926c0)
    /usr/local/go/src/net/http/client.go:320 +0x17c
created by net/http.setRequestCancel
    /usr/local/go/src/net/http/client.go:330 +0x287

goroutine 220550 [select]:
net/http.(*persistConn).readLoop(0xc4218f0c60)
    /usr/local/go/src/net/http/transport.go:1599 +0x9ec
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1117 +0xa35

goroutine 216169 [IO wait]:
net.runtime_pollWait(0x7f6373700fb0, 0x72, 0x221)
    /usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc4212fdb18, 0x72, 0xa85c40, 0xa81350)
    /usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc4212fdb18, 0xc42393c000, 0x1000)
    /usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc4212fdab0, 0xc42393c000, 0x1000, 0x1000, 0x0, 0xa85c40, 0xa81350)
    /usr/local/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc42300aa38, 0xc42393c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:181 +0x70
net/http.(*persistConn).Read(0xc422102c60, 0xc42393c000, 0x1000, 0x1000, 0xc423d30820, 0xc42002a800, 0xc424422b80)
    /usr/local/go/src/net/http/transport.go:1316 +0x14b
bufio.(*Reader).fill(0xc4220598c0)
    /usr/local/go/src/bufio/bufio.go:97 +0x117
bufio.(*Reader).Peek(0xc4220598c0, 0x1, 0xc422059a40, 0xc420535c80, 0xc422059700, 0x60, 0xc420535c60)
    /usr/local/go/src/bufio/bufio.go:129 +0x67
net/http.(*persistConn).readLoop(0xc422102c60)
    /usr/local/go/src/net/http/transport.go:1474 +0x196
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1117 +0xa35

goroutine 213706 [select]:
net/http.(*persistConn).writeLoop(0xc42321d200)
    /usr/local/go/src/net/http/transport.go:1704 +0x43a
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1118 +0xa5a

goroutine 214335 [select]:
net/http.(*persistConn).writeLoop(0xc4230d7560)
    /usr/local/go/src/net/http/transport.go:1704 +0x43a
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1118 +0xa5a

goroutine 221651 [IO wait]:
net.runtime_pollWait(0x7f63736eff38, 0x72, 0x254)
    /usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc422449b18, 0x72, 0xa85c40, 0xa81350)
    /usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc422449b18, 0xc423d93000, 0x1000)
    /usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc422449ab0, 0xc423d93000, 0x1000, 0x1000, 0x0, 0xa85c40, 0xa81350)
    /usr/local/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc421eb21b8, 0xc423d93000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:181 +0x70
net/http.(*persistConn).Read(0xc422e5b440, 0xc423d93000, 0x1000, 0x1000, 0x8cc888, 0x0, 0xc42347bb80)
    /usr/local/go/src/net/http/transport.go:1316 +0x14b
bufio.(*Reader).fill(0xc422f2c960)
    /usr/local/go/src/bufio/bufio.go:97 +0x117
bufio.(*Reader).Peek(0xc422f2c960, 0x1, 0xc422f2d6e0, 0xc420b8ec80, 0xc42180ff00, 0x60, 0xc420b8ec60)
    /usr/local/go/src/bufio/bufio.go:129 +0x67
net/http.(*persistConn).readLoop(0xc422e5b440)
    /usr/local/go/src/net/http/transport.go:1474 +0x196
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1117 +0xa35

goroutine 221670 [IO wait]:
net.runtime_pollWait(0x7f63736ff7b0, 0x72, 0x259)
    /usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc422449b88, 0x72, 0xa85c40, 0xa81350)
    /usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc422449b88, 0xc423f28000, 0x1000)
    /usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc422449b20, 0xc423f28000, 0x1000, 0x1000, 0x0, 0xa85c40, 0xa81350)
    /usr/local/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc423eac3c8, 0xc423f28000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:181 +0x70
net/http.(*persistConn).Read(0xc4235930e0, 0xc423f28000, 0x1000, 0x1000, 0xa82dc0, 0xc42000e0b0, 0xc423be7b80)
    /usr/local/go/src/net/http/transport.go:1316 +0x14b
bufio.(*Reader).fill(0xc422bb70e0)
    /usr/local/go/src/bufio/bufio.go:97 +0x117
bufio.(*Reader).Peek(0xc422bb70e0, 0x1, 0xc422bb7320, 0xc424191480, 0xc4237a4820, 0x0, 0xc422ac9138)
    /usr/local/go/src/bufio/bufio.go:129 +0x67
net/http.(*persistConn).readLoop(0xc4235930e0)
    /usr/local/go/src/net/http/transport.go:1474 +0x196
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1117 +0xa35

goroutine 213543 [select]:
net/http.(*persistConn).readLoop(0xc424795c20)
    /usr/local/go/src/net/http/transport.go:1599 +0x9ec
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1117 +0xa35

goroutine 220529 [select]:
net/http.(*persistConn).roundTrip(0xc4215fd680, 0xc4212c5c80, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1898 +0x974
net/http.(*Transport).RoundTrip(0xc420847b30, 0xc421248800, 0xc420847b30, 0xed15b2cf4, 0x16cd81f7)
    /usr/local/go/src/net/http/transport.go:391 +0x74c
net/http.send(0xc421248700, 0xa84080, 0xc420847b30, 0xed15b2cf4, 0x16cd81f7, 0xab7060, 0x0, 0x8, 0xc4213a2a80, 0x40f258)
    /usr/local/go/src/net/http/client.go:249 +0x162
net/http.(*Client).send(0xc42075ed20, 0xc421248700, 0xed15b2cf4, 0x16cd81f7, 0xab7060, 0xc4213a2a80, 0x0, 0x1, 0x4084fe)
    /usr/local/go/src/net/http/client.go:173 +0x108
net/http.(*Client).Do(0xc42075ed20, 0xc421248700, 0xa, 0x8b359b, 0x5)
    /usr/local/go/src/net/http/client.go:595 +0x254
github.com/myCompanyName/gojsonrpc.(*Client).sendJsonRequest(0xc42078c6e0, 0xc4215fd320, 0xbb, 0x11a, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/ubuntu/go/src/github.com/myCompanyName/gojsonrpc/client.go:45 +0x2f2
github.com/myCompanyName/gojsonrpc.(*Client).Run(0xc42078c6e0, 0x8be6b2, 0x1d, 0x838780, 0xc420789ce0, 0x7fb2e0, 0xc4212c5b40, 0x1, 0xe)
    /home/ubuntu/go/src/github.com/myCompanyName/gojsonrpc/client.go:72 +0x219
github.com/myCompanyName/app/connectors/vendor.(*Connector).ReadStatus(0xc42078aa30, 0xc4212c5a60, 0xd, 0xc4212c5a6e, 0x7, 0xc4212c59c0, 0x1d, 0x0, 0x0, 0x0, ...)
    /home/ubuntu/go/src/github.com/myCompanyName/app/connectors/vendor/tickets.go:52 +0x386
github.com/myCompanyName/app/obscuredPkgName/connabslayer.(*ConnAbsLayer).ReadMetrics(0xc42075ac40, 0xc4212c5a60, 0x15, 0xc4212c59c0, 0x1d, 0xc42121eb30, 0xf)
    /home/ubuntu/go/src/github.com/myCompanyName/app/obscuredPkgName/connabslayer/connector_abstraction_layer.go:145 +0xdc
github.com/myCompanyName/app/obscuredPkgName/updater/plugins.(*UpdaterPlugin).SyncPluggedData(0xc42078c820, 0xc4208b9ef0, 0x28, 0xf)
    /home/ubuntu/go/src/github.com/myCompanyName/app/obscuredPkgName/updater/plugins/vendor_plugin.go:69 +0x1e1
github.com/myCompanyName/app/obscuredPkgName/updater.(*Updater).syncData.func1(0xc420763a00, 0xc4208b9ef0, 0x28, 0xc4201824f0, 0xf)
    /home/ubuntu/go/src/github.com/myCompanyName/app/obscuredPkgName/updater/updater.go:127 +0xf6
created by github.com/myCompanyName/app/obscuredPkgName/updater.(*Updater).syncData
    /home/ubuntu/go/src/github.com/myCompanyName/app/obscuredPkgName/updater/updater.go:131 +0x17b

// AND SO ON

Solution:

The problem was elsewhere, as visible in the comments below, but the question might be useful for somebody else asking the same.

Related / possible duplicate of [Do we need to close the response object if an error occurs while calling http.Get(url)?](https://stackoverflow.com/questions/32818472/do-we-need-to-close-the-response-object-if-an-error-occurs-while-calling-http-ge/32819910#32819910) — icza, Sep 26 '17 at 10:04
1. Yes as stated by the doc. But if err!=nil you can ignore the body. 2. No. see the doc you cited. — Volker, Sep 26 '17 at 10:05
That is only part of the stack trace, what error are you actually getting? What version of Go are you using? The stack trace doesn't look like it's from a recent release. — JimB, Sep 26 '17 at 13:09
it's 1.8, I might have found the issue and it has nothing to do with the http client, it might be a concurrent map write that must be synced, the panic is not handled and every goroutine gets printed after the `fatal error: concurrent map read and map write` — dev_mush, Sep 26 '17 at 14:22
@dev_mush: that's not a panic that can be handled, it's a fatal error because you have a data race (and you should update your Go version, there have been some security fixes in the 1.8 cycle since then, and go1.9 has been out for a while) — JimB, Sep 26 '17 at 14:25
@JimB right! thanks a lot for this, I didn't know it couldn't be handled and will look into it right away, and will obviously proceed to synchronise the access to the map! (and update go version on the server) — dev_mush, Sep 26 '17 at 14:28

score 5 · Accepted Answer · edited Sep 26 '17 at 14:26

Yes, they can both be non-nil in one situation, apparently. From the source, we see:

        if err != nil {
            // Special case for Go 1 compatibility: return both the response
            // and an error if the CheckRedirect function failed.
            // See https://golang.org/issue/3795
            // The resp.Body has already been closed.
            ue := uerr(err)
            ue.(*url.Error).URL = loc
            return resp, ue
        }

Every other return is returning either a response and nil err, or a nil response and non-nil error.
So for the questions above:

Yes
Can't happen, for the only case in which both a body and an error are non-nil, the body has already been closed

so in this case anyway the body gets closed, which means that my problem can't be related to bodies not closed by the defer... — dev_mush, Sep 26 '17 at 14:08

When using go http client Do method, can the httpResponse and the error be not nil at the same time?

1 Answers1