Python use CONNECT http verbs, header manipulation, and TLS/SSL

Question

In Python, how do I use the CONNECT http verb and do http header manipulation, and TLS/SSL?

Unfortunately requests/urllib don't support this.

Trying to avoid manually doing this with openssl + sockets.

Answer 1

You're reading a blog post from 2013, which contains a link to a urllib3 Github issue, which was fixed about a month later, and support was added to requests shortly afterward, and is there in all 2.x versions.

So, you're looking for a workaround for a problem that was solved almost 5 years ago.

To use an HTTPS proxy, you just configure it the same way as an HTTP proxy:

proxies = {
    'https': 'http://10.10.1.10:12345',
}
page = requests.get('https://example.org', proxies=proxies)

If you, e.g., run nc -kl 1080 on 10.10.1.10, you'll see this:

CONNECT example.org:443 HTTP/1.0

And if you run an actual HTTPS proxy there, it will just work.

---

You also claim that urllib doesn't handle HTTPS proxies, but it always has. It's slightly more painful to set up, but still not that hard:

ph = urllib.request.ProxyHandler({'https': '192.168.42.100:1080'})
op = urllib.request.build_opener(p)
page = op.open('https://example.com')

… or, if you want to use it for everything rather than a single request:

ph = urllib.request.ProxyHandler({'https': '192.168.42.100:1080'})
urllib.request.install_opener(urllib.request.build_opener(p))
page = urllib.request.open('https://example.com')

---

And of course if you have your default proxy settings configured in the appropriate way for your platform, you don't even need to do this much; both requests and urllib will just use them.