How to local test gae golang urlfetch through proxy?

Question

My urlfetch client works fine when deployed to appspot. But local testing (dev_appserver.py) through proxy has issue. I can't find any way to set proxy for urlfetch.Transport.

How do you test urlfetch behind proxy locally?

Answer 1

http.DefaultTransport and http.DefaultClient are not available in App Engine. See https://developers.google.com/appengine/docs/go/urlfetch/overview

Got this error message when testing PayPal OAuth on GAE dev_appserver.py (works in production when compiled)

const url string = "https://api.sandbox.paypal.com/v1/oauth2/token"
const username string = "EOJ2S-Z6OoN_le_KS1d75wsZ6y0SFdVsY9183IvxFyZp"
const password string = "EClusMEUk8e9ihI7ZdVLF5cZ6y0SFdVsY9183IvxFyZp"

client := &http.Client{}        

req, _ := http.NewRequest("POST", url, strings.NewReader("grant_type=client_credentials"))
req.SetBasicAuth(username, password)
req.Header.Set("Accept", "application/json")
req.Header.Set("Accept-Language", "en_US")
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")

resp, err := client.Do(req)

As you can see, Go App Engine breaks http.DefaultTransport (GAE_SDK/goroot/src/pkg/appengine_internal/internal.go, line 142, GAE 1.7.5)

type failingTransport struct{}
func (failingTransport) RoundTrip(*http.Request) (*http.Response, error) {
    return nil, errors.New("http.DefaultTransport and http.DefaultClient are not available in App Engine. " +
        "See https://developers.google.com/appengine/docs/go/urlfetch/overview")
}

func init() {
    // http.DefaultTransport doesn't work in production so break it
    // explicitly so it fails the same way in both dev and prod
    // (and with a useful error message)
    http.DefaultTransport = failingTransport{}
}

This solved it to me with Go App Engine 1.7.5

    transport := http.Transport{}

    client := &http.Client{
        Transport: &transport,
    }       

    req, _ := http.NewRequest("POST", url, strings.NewReader("grant_type=client_credentials"))
    req.SetBasicAuth(username, password)
    req.Header.Set("Accept", "application/json")
    req.Header.Set("Accept-Language", "en_US")
    req.Header.Set("Content-Type", "application/x-www-form-urlencoded")

Answer 2

This is just a guess, but did you try setting the proxy variables

In a Unix or Windows environment, set the http_proxy, or ftp_proxy environment variables to a URL that identifies the proxy server before starting the Python interpreter. For example (the '%' is the command prompt):

% http_proxy="http://www.someproxy.com:3128"

% export http_proxy

Answer 3

if you are using the default proxy then the transport is implemented as

var DefaultTransport RoundTripper = &Transport{Proxy: ProxyFromEnvironment} 

setting the environment variable when launching go should solve the problem.

See also this other question: How do I configure Go to use a proxy?

Answer 4

The urlfetch package itself does not honor proxy settings, even in development, because it's not actually doing the URL fetch itself: it sends a request to the (possibly development) app server and asks it to do the fetch. I don't have the source of dev_appserver.py handy, but it should honor the standard proxy variables:

export http_proxy='http://user:pass@1.2.3.4:3210/'

If you do that before you start dev_appserver.py, it will probably just work.

If the above does not work, you should file an issue and then use the following workaround:

func client(ctx *appengine.Context) *http.Client {
    if appengine.IsDevAppServer() {
        return http.DefaultClient
    }
    return urlfetch.Client(ctx)
}

This will use the urlfetch API on the production appserver but use the standard net/http client otherwise, which does honor the http_proxy environment variable.