HTTP Client doesn't return response when status is 301 without a Location header

Question

When I make a request with Go's net/http/client package, and the server responds with 301 or 302 status code, but doesn't provide a Location header, we get an error object, and no response object. Example ReplIt session: https://replit.com/@sharat87/RedirectWithoutLocationGo#main.go. Including source here in case that session goes away in the future:

package main

import (
  "log"
    "net/http"
)

func main() {
    client := &http.Client{}
    resp, err := client.Get("https://httpbun.com/status/301")
    if err != nil {
        log.Println("Error getting", err)
    } else {
        log.Println(resp.Status)
    }
}

This prints the error message: Error getting Get "https://httpbun.com/status/301": 301 response missing Location header.

However, the Location header in 301 or 302 status responses is optional. Servers are not required to provide it, and clients are not obliged to redirect to it. In fact, in my case, I need to look at the other headers as well as the response body for what I need. But when the Location header is missing, Go just returns an error and discards the whole response. For example, curl doesn't throw an error in this case, and actually responds with the other headers:

$ curl -v https://httpbun.com/status/301
[... TLS logs redacted]
> GET /status/301 HTTP/1.1
> Host: httpbun.com
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Server: nginx/1.18.0 (Ubuntu)
< Date: Sun, 25 Jul 2021 02:18:53 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 21
< Connection: keep-alive
< X-Powered-By: httpbun
<

I dug into the source for the net/http/client module to try and see what might be happening. The error message comes up from this line in the source. Here, for the redirected request, we first check if a Location header is available, and if not, we are returning an error. This is very limiting especially because the spec doesn't describe this to be an error condition and in fact, lists a legitimate reason for doing this (i.e., the server doesn't know where to redirect to) (reference, links to sources).

So, my question is, is this intended? Am I missing something here? Would a PR to change this be invited? (Asking here because looks like Golang repo issues on GitHub are reserved for proposals?).

I'd love any advise here since I'm essentially blocked here.

Edit: Fixed in https://github.com/golang/go/issues/49281.

Answer 1

Create a transport wrapper that fixes up the response for the client. The following example sets the location header when the header is missing. Another option is to change the status code.

type locFix struct {
    http.RoundTripper
}

func (lf locFix) RoundTrip(req *http.Request) (*http.Response, error) {
    resp, err := lf.RoundTripper.RoundTrip(req)
    if err != nil {
        return resp, err
    }   
    if resp.StatusCode == 301 || resp.StatusCode == 302 {
        if resp.Header.Get("Location") == "" {      
            resp.Header.Set("Location", "http://example.com")
        }
    }
    return resp, err
}

Use a http.Client with the wrapped transport. Here’s how to create a wrapper for the net/http default client:

client := *http.DefaultClient
t := client.Transport
if t == nil {
    t = http.DefaultTransport
}
client.Transport = locFix{t}

resp, err = client.Get("https://httpbun.com/status/301")

Answer 2

The default client follows redirects. You can bypass this by using RoundTrip directly:

package main

import (
   "fmt"
   "net/http"
)

func main() {
   req, err := http.NewRequest("GET", "http://httpbun.com/status/301", nil)
   if err != nil {
      panic(err)
   }
   res, err := new(http.Transport).RoundTrip(req)
   if err != nil {
      panic(err)
   }
   defer res.Body.Close()
   fmt.Printf("%+v\n", res)
}