How do you use an HTTP/HTTPS proxy with boto3?

Question

On the old boto library is was simple enough to use the proxy, proxy_port, proxy_user and proxy_pass parameters when you open a connection. However, I could not find any equivalent way of programmatically define the proxy parameters on boto3. :(

At the moment, I'm having to use something like: `import os ;os.environ["HTTP_PROXY"] = "http://yourproxy.com:1234"; os.environ["HTTPS_PROXY"] = "https://yourproxy1.com:1234"` — Mani, Nov 03 '15 at 11:31
I did not find any mention on the documentation that this would work. For example, on boto using `HTTPS_PROXY` would not work, only `HTTP_PROXY`. — asieira, Nov 03 '15 at 12:22
Anyway, opened an issue at the boto3 GitHub repository: https://github.com/boto/boto3/issues/338 — asieira, Nov 03 '15 at 12:23
Seems like the devs thinks that setting the environment is a worthy replacement (I don't). — Mani, Nov 10 '15 at 08:25
When using environment variables (related but different from this question) `HTTPS_PROXY` works for me but not `HTTP_PROXY` — Alex R, Mar 19 '20 at 00:34

cblakkan · Accepted Answer · 2018-03-12T16:57:38.697

37

As of at least version 1.5.79, botocore accepts a proxies argument in the botocore config.

e.g.

import boto3
from botocore.config import Config

boto3.resource('s3', config=Config(proxies={'https': 'foo.bar:3128'}))

boto3 resource https://boto3.readthedocs.io/en/latest/reference/core/session.html#boto3.session.Session.resource

botocore config https://botocore.readthedocs.io/en/stable/reference/config.html#botocore.config.Config

edited Mar 12 '18 at 16:57

answered Aug 03 '17 at 18:39

cblakkan

486
4
5

2

I'm using boto3==1.4.6, botocore==1.6.6, but this does not seem to be working for me. Could you please provide a full example loading a file into a bucket, or something similar? – albarji Aug 24 '17 at 10:26
Since AWS uses HTTPS for all endpoints, try this configuration: `boto3.resource('s3', config=Config(proxies={'https': 'foo.bar:3128'}))` (note that it's http**s**) – Brad Sep 28 '17 at 00:02
2

I had the same issue with STS. I've put that config into client and it worked, In case someone else need: `conn = boto3.client('sts', config=Config(proxies={'http': 'myproxy', 'https': 'myproxy'}))` – PedroMVM Aug 13 '19 at 12:02

score 18 · Answer 2 · edited Jun 09 '20 at 11:18

18

If you user proxy server does not have a password try the following:

import os
os.environ["HTTP_PROXY"] = "http://proxy.com:port"
os.environ["HTTPS_PROXY"] = "https://proxy.com:port"

if you user proxy server has a password try the following:

import os
os.environ["HTTP_PROXY"] = "http://user:password@proxy.com:port"
os.environ["HTTPS_PROXY"] = "https://user:password@proxy.com:port"

edited Jun 09 '20 at 11:18

Ron

22,128
31
108
206

answered Nov 03 '15 at 14:09

user881703

1,111
3
19
38

Although ensure you correctly include quotations around your envirion values. This answer has them missing at the beginning of the proxy – mightymephisto Apr 07 '16 at 10:15
This works, but changing an environment variable is troublesome. If along your program you need to perform an http request to other server, such request will get routed through the s3 proxy server, which is not what you want. You can kind of solve this by restoring both env variables to their original values after you are finished querying s3, but I wish there was a better solution. – albarji Aug 24 '17 at 10:13
@albarji, it depends on how you design your application, it is a basic principle to environment values isolate from the main application. Then you may have s3_proxy.config, serverA_proxy.config – user881703 Sep 27 '21 at 07:00

score 3 · Answer 3 · edited Aug 13 '19 at 01:25

Apart from altering the environment variable, I'll present what I found in the code.

Since boto3 uses botocore, I had a look through the source code:

https://github.com/boto/botocore/blob/66008c874ebfa9ee7530d944d274480347ac3432/botocore/endpoint.py#L265

From this link, we end up at:

    def _get_proxies(self, url):
        # We could also support getting proxies from a config file,
        # but for now proxy support is taken from the environment.
        return get_environ_proxies(url)

...which is called by proxies = self._get_proxies(final_endpoint_url) in the EndpointCreator class.

Long story short, if you're using python2 it will use the getproxies method from urllib2 and if you're using python3, it will use urllib3.

get_environ_proxies is expecting a dict containing {'http:' 'url'} (and I'm guessing https too).

You could always patch the code, but that is poor practice.

score 2 · Answer 4 · answered Mar 24 '16 at 20:21

This is one of the rare occasions when I would recommend monkey-patching, at least until the Boto developers allow connection-specific proxy settings:

import botocore.endpoint
def _get_proxies(self, url):
    return {'http': 'http://someproxy:1234/', 'https': 'https://someproxy:1234/'}
botocore.endpoint.EndpointCreator._get_proxies = _get_proxies
import boto3

How do you use an HTTP/HTTPS proxy with boto3?

4 Answers4

Linked