6

Is there a way to specify a proxy server when using urlfetch on Google App Engine?

Specifically, every time I make a call using urlfetch, I want GAE to go through a proxy server. I want to do this on production, not just dev.

I want to use a proxy because there are problems with using google's outbound IP addresses (rate limiting, no static outbound IP, sometimes blacklisted, etc.). Setting a proxy is normally easy if you can edit the http message itself, but GAE's API does not appear to let you do this.

speedplane
  • 15,673
  • 16
  • 86
  • 138
  • How do you think setting a proxy will help with rate limiting etc? – Daniel Roseman Jan 10 '12 at 08:01
  • @DanielRoseman - if the 3rd party is limiting requests on a per-IP basis, using a proxy would help. Twitter, for example, limit requests to 150/hour/IP address. – Ben Parsons Jan 10 '12 at 09:09
  • @DanielRoseman BenP is right. There are thousands of apps on each GAE IP address and those limits get used up instantly (thanks for the downvote btw). – speedplane Jan 10 '12 at 13:37

2 Answers2

3

You can always roll your own:

  1. In case of fixed destination: just setup a fixed port forwarding on a proxy server. Then send requests from GAE to proxy. If you have multiple destinations, then set forwarding on separate ports, one for each destination.

  2. In case of dynamic destination (too much to handle via fixed port forwarding), your GAE app adds a custom http header (X-Something) containing final destination and then connects to custom proxy. Custom proxy inspects this field and forwards the request to the destination.

Peter Knego
  • 79,991
  • 11
  • 123
  • 154
  • I'm in the same situation as @speedplane, this was quite helpful. Having never done this before, is there an existing proxy server you'd recommend for doing something like this (like say Squid)? – Rob Boyle Jun 08 '12 at 20:33
  • 1
    Hi Peter, are you aware of a proxy server system that can scale as nicely as GAE? – speedplane Aug 04 '12 at 19:21
1

We ran into this issue and reached out to Google Cloud support. They suggested we use Google App Engine flexible with some app.yaml settings, custom network, and an ip-forwarding NAT gateway instance.

This didn't work for us because many core features from App Engine Standard are not implemented in App Engine Flexible. In essence we would need to rewrite our product.

So, to make applicable URL fetch requests appear to have a static IP we made a custom proxy: https://github.com/csgactuarial/app-engine-proxy

For redundancy reasons, I suggest implementing this as a multi region, multi zone, load balanced system.

chmoder
  • 876
  • 10
  • 19
  • Thanks very helpful, apparently we had implemented a similar proxy server to you except we were missing the `headers.host` configuration when creating `proxy.web`. – ahong Apr 27 '20 at 04:44