21

I'm running a Python script that uses the requests package for making web requests. However, the web requests go through a proxy with a self-signed cert. As such, requests raise the following Exception:

requests.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')],)",)

I know that SSL validation can be disabled in my own code by passing verify=False, e.g.: requests.get("https://www.google.com", verify=False). I also know that if I had the certificate bundle, I could set the REQUESTS_CA_BUNDLE or CURL_CA_BUNDLE environment variables to point to those files. However, I do not have the certificate bundle available.

How can I disable SSL validation for external modules without editing their code?

Moshe
  • 9,283
  • 4
  • 29
  • 38

2 Answers2

53

Note: This solution is a complete hack.

Short answer: Set the CURL_CA_BUNDLE environment variable to an empty string.

Before:

$ python
import requests
requests.get('http://www.google.com')
<Response [200]>

requests.get('https://www.google.com')
...
File "/usr/local/lib/python2.7/site-packages/requests-2.17.3-py2.7.egg/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')],)",)

After:

$ CURL_CA_BUNDLE="" python
import requests
requests.get('http://www.google.com')
<Response [200]>

requests.get('https://www.google.com')
/usr/local/lib/python2.7/site-packages/urllib3-1.21.1-py2.7.egg/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning)
<Response [200]>

How it works

This solution works because Python requests overwrites the default value for verify from the environment variables CURL_CA_BUNDLE and REQUESTS_CA_BUNDLE, as can be seen here:

if verify is True or verify is None:
verify = (os.environ.get('REQUESTS_CA_BUNDLE') or
    os.environ.get('CURL_CA_BUNDLE'))

The environment variables are meant to specify the path to the certificate file or CA_BUNDLE and are copied into verify. However, by setting CURL_CA_BUNDLE to an empty string, the empty string is copied into verify and in Python, an empty string evaluates to False.

Note that this hack only works with the CURL_CA_BUNDLE environment variable - it does not work with the REQUESTS_CA_BUNDLE. This is because verify is set with the following statement:

verify = (os.environ.get('REQUESTS_CA_BUNDLE') or os.environ.get('CURL_CA_BUNDLE'))

It only works with CURL_CA_BUNDLE because '' or None is not the same as None or '', as can be seen below:

print repr(None or "")
# Prints: ''
print repr("" or None )
# Prints: None
Moshe
  • 9,283
  • 4
  • 29
  • 38
  • 4
    I've successfully used this technique on Unix, but cannot get it to work on Windows – Craig Brett Apr 15 '19 at 09:24
  • Have you tried `set CURL_CA_BUNDLE="" ` or `setx CURL_CA_BUNDLE="" ` on windows machine? – SilentGuy May 29 '19 at 16:59
  • Also not on Mac :( – Barney Szabolcs Oct 24 '19 at 14:16
  • 1
    Also with `set CURL_CA_BUNDLE=""` not working on Windows for me – Pieter Meiresone Feb 17 '20 at 10:50
  • Interesting thing (in my case Linux Ubuntu): this works only if I start python command with `CURL_CA_BUNDLE=""` and in same line put `python`. It does not work if I put `CURL_CA_BUNDLE=""` and press `` then call Python in next command line. – David Jul 15 '20 at 06:27
  • 2
    This is amazing in its simplicity and hackiness :) Confirming this works as described on macOS 10.14 with Python 3.7/3.8. – BrianC Aug 06 '20 at 22:11
  • 2
    @David: That's how variables work in Bash and similar shells. If you just do `CURL_CA_BUNDLE=""`, that only sets the variable within the context of that specific shell or for the command that follows on the same line. If you use `export CURL_CA_BUNDLE=""`, then it will also apply to any new processes started from that shell, like Python (but still not ones in completely separate processes). – Soren Bjornstad Oct 13 '20 at 14:23
  • 4
    This no longer works with requests 2.28.0 or newer, since it was considered a bug, see https://github.com/psf/requests/issues/6071... – Kenneth Hoste Jun 09 '22 at 17:36
4

I saw this hack only due to some trouble with my private CA.

The given hack with CURL_CA_BUNDLE='' will not longer work in 2022 with next minor release of requests (that means 2.28 ?).

Please refer to GH issue 6071 which was fixed 6 days before in Feb. 2022

psytester
  • 194
  • 1
  • 5