using python-requests with cntlm corporate proxy

Question

I got some python applications I'd like to run behind a corporate proxy. I'm using cntlm and I think it's configured properly so far because wget, curl and pip and so on are working pretty well. i.e.:

sudo wget http://apple.com

Console-Output:

andre@VirtualBox:~$ wget apple.com
--2018-04-04 10:38:55--  http://apple.com/
Connecting to 127.0.0.1:3128... connected.
Proxy request sent, awaiting response... 301 Moved Permanently
Location: https://www.apple.com/ [following]
--2018-04-04 10:38:56--  https://www.apple.com/
Connecting to 127.0.0.1:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: 45704 (45K) [text/html]
Saving to: ‘index.html.3’

index.html.3        100%[===================>]  44,63K  --.-KB/s    in 0s      

2018-04-04 10:38:56 (312 MB/s) - ‘index.html.3’ saved [45704/45704]

is working - where this:

import requests
url = 'https://apple.com'
session = requests.session()
r = session.get(url)
print(r.text)

Console-Output:

Traceback (most recent call last):
  File "/home/andre/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 595, in urlopen
    self._prepare_proxy(conn)
  File "/home/andre/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 816, in _prepare_proxy
    conn.connect()
  File "/home/andre/.local/lib/python3.5/site-packages/urllib3/connection.py", line 294, in connect
    self._tunnel()
  File "/usr/lib/python3.5/http/client.py", line 832, in _tunnel
    message.strip()))
OSError: Tunnel connection failed: 407 Proxy Authentication Required

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/andre/.local/lib/python3.5/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/home/andre/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/home/andre/.local/lib/python3.5/site-packages/urllib3/util/retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='apple.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required',)))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/andre/PycharmProjects/Notifications/de/seandre/rest/requesttest.py", line 5, in <module>
    r = session.get(url)
  File "/home/andre/.local/lib/python3.5/site-packages/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/home/andre/.local/lib/python3.5/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/andre/.local/lib/python3.5/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/home/andre/.local/lib/python3.5/site-packages/requests/adapters.py", line 502, in send
    raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPSConnectionPool(host='apple.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required',)))

is not working.

Many thanks in advance!

score 0 · Answer 1 · answered Jan 24 '18 at 11:42

You should set environment variable HTTP_PROXY

export HTTP_PROXY='http://mydomain\username:12345@servername.org:3128'

And try this example code.

url = 'https://example.com/example'
req = urllib.request.Request(
url,
data=None,
headers={'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36'})
proxy = urllib.request.ProxyHandler({'https': os.environ['HTTP_PROXY']})
auth = urllib.request.HTTPBasicAuthHandler()
opener = urllib.request.build_opener(proxy, auth, urllib.request.HTTPHandler)
urllib.request.install_opener(opener)
data = urllib.request.urlopen(req)

But for me very usefull cntlm proxy

thanks very much! But the thing is that I'm already using cntlm with ubuntu and it's working perfectly fine when I'm using wget or curl but not with Python Scripts. I've set local env. variable to HTTP_PROXY="http://localhost:3128/" — Andrew, Jan 25 '18 at 13:22

using python-requests with cntlm corporate proxy

1 Answers1