11

In the AppEngine developer appserver I am getting an error like this:

SSLCertificateError: Invalid and/or missing SSL certificate for URL ...

when I am making a fetch like this to an https server with a self-signed certificate (almost always localhost port-forwarded over ssh to a vm):

result = urlfetch.fetch(url=url, method=method, payload=payload,
                        deadline=DEADLINE, validate_certificate=None)

One would not expect SSL failures for invalid certificates where validate_certificate is False, though this is quite possibly a side-effect of the 2.7.9 policy in Python to always validate ssl certificates.

Note that passing False (instead of None) for validate_certificate does not work either.

This problem happens on Python 2.7.9-10 via Homebrew/XCode on OS X 10.10.2-4 with AppEngine 1.9.18 through 1.19.26.

There are issues (e.g. 12096) about this on Google App Engine, but I am looking for a workaround.

Here's what I've tried to work around this:

  1. Add the certificate to the Mac's login keychain (works in the browser, not from Python)

  2. Add the certificate to app-engine-python/lib/cacerts/cacerts.txt and/or ./lib/cacerts/urlfetch_cacerts.txt (though this probably requires turning verification on for it to work, since that appears to be the only case where they are used) with e.g.

    $ echo >> /usr/local/share/app-engine-python/lib/cacerts/urlfetch_cacerts.txt

    $ openssl x509 -subject -in server.crt >> /usr/local/share/app-engine-python/lib/cacerts/urlfetch_cacerts.txt

  3. Disable ssl HTTPs checking with the PEP-0476 workaround i.e.

    ssl._create_default_https_context = ssl._create_unverified_context

    at or after import ssl (around line 1149) of google/appengine/dist27/python_std_lib/httplib.py

This is particularly problematic on Mac since downgrading as of XCode 7/OS X El Capital is no longer a practical option.

A preferable workaround would not involve monkey-patching the AppEngine code proper every time the development appserver is updated.

EDIT

Note that the Mac builtin OpenSSL certificates are stored in /System/Library/OpenSSL, which is protected with SIP/rootlessness, which frankly is a pain to muck with and a worthwhile feature to keep if we can.

I have verified that the certificate validates by using openssl s_client -connect localhost:7500 -CAfile server.pem.

It's been added to the Keychain and to /usr/local/etc/openssl/certs with the hash.# format where the hash comes from openssl x509 -subject_hash -in server.pem (or the homebrew ssl, namely /usr/local/Cellar/openssl/1.0.2d_1/bin/openssl). In which case /usr/local/Cellar/openssl/1.0.2d_1/bin/openssl s_client -connect localhost:7500 verifies the certificate (but python still does not).

I have tried using the homebrew version of python and openssl, but to no avail. Running the following in Python seems to always fail;

./pve/bin/python -c "import requests; requests.get('https://localhost:7500')"

This also fails where SSL_CERT_FILE is set to the server's certificate (i.e. for added measure one might expect it to work since the openssl command essentially works like this), and also fails where SSL_CERT_PATH is set to /usr/local/etc/openssl/certs.

Note, pve is a virtual env where help(ssl) shows a FILE of /usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py

Further verifying that homebrew Python's _ssl.so links to homebrew's openssl I ran:

xcrun otool -L /usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_ssl.so

which returns

./Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_ssl.so:

/usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)

/usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)

/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1225.1.1)

If one runs brew info openssl it remarks under CAVEATS:

A CA file has been bootstrapped using certificates from the system keychain. To add additional certificates, place .pem files in /usr/local/etc/openssl/certs

but clearly for some reason python is not using homebrew's openssl algorithm for finding certificates.

So I remain at a loss as to why Python standard library is not validating certificates that are in the OpenSSL directory specified in the documents as well as the Keychain (in both .pem and .p12 formats, with "always trust" for Secure Sockets Layer (SSL)).

Community
  • 1
  • 1
Brian M. Hunt
  • 81,008
  • 74
  • 230
  • 343

4 Answers4

8

This is a dev_appserver bug caused by a httplib.HTTPSConnection behavior change (certificate check turned on by default) in some recent Python release (I belive 2.7.9).

As the bug is in internal dev_appserver code (file google_appengine/google/appengine/api/urlfetch_stub.py of the appengine SDK) that is run independently of the tested application, there is no way to make a fix that will survive a SDK update.

The only permanent workaround I can think of will be to enable validate_certificate and add CA certificate to the urlfetch_cacerts.txt file. As a temporary fix, you can patch urlfetch_stub.py with workaround #3.

spenibus
  • 4,339
  • 11
  • 26
  • 35
Alex
  • 474
  • 3
  • 10
  • Thanks @Alex. Should it not be possible to add the certificate to the Mac's system, that Python's Httplib will then use? – Brian M. Hunt Sep 30 '15 at 13:19
  • Brian, I'm not very familiar with how python ssl lib works on Mac, I would check [ssl doc page](https://docs.python.org/2/library/ssl.html#ssl-contexts) and see what is the default path string: `ssl.get_default_verify_paths()`. May be there is a way to use [env variable to set a path to yours certificate file](https://www.python.org/dev/peps/pep-0476/#trust-database). – Alex Sep 30 '15 at 18:43
  • [Here is the relevant doc](http://gagravarr.org/writing/openssl-certs/others.shtml) on how to add self-signed cert for openssl. – Alex Oct 02 '15 at 22:46
  • Thanks Alex. I've tried, but not yet succeeded, with adding my certificates to the keychain. Here's another description of how to do so though, that you perhaps might be interested in including in your answer: http://apple.stackexchange.com/questions/8993 – Brian M. Hunt Oct 04 '15 at 01:11
  • So I've added the `.p12`, `csr`, and `crt` to my key chain, and the certificate is trusted by my browsers, but not by Python. So the only solution that has worked so far has been modifying `urlfetch_stub.py`. – Brian M. Hunt Oct 04 '15 at 13:51
  • You are saying you installed MacOS system certificate, did you try to install OpenSSL framework cert as described in [this section](http://gagravarr.org/writing/openssl-certs/others.shtml#ca-openssl)? I guess openssl library is not using MacOS system certificates but only it's own. – Alex Oct 05 '15 at 21:20
  • Thanks Alex. As you can see from my edit, the problem may be broader than Appserver (though Appserver is peculiarly vulnerable by not having a workaround to the `validate_certificate` param to `urlfetch` not being respected). – Brian M. Hunt Oct 06 '15 at 02:25
  • Just thoghts... Is python installed with brewed ssl? `brew install python --with-brewed-openssl` – Alex Oct 06 '15 at 08:25
  • Thanks Alex. I've tried clean installing python with `--with-brewed-openssl` (though there's no evidence that it's being used/respected in the code, that I can see), as well as `brew link openssl --force`. Neither appears to have any effect -- and it's not clear why the certificate fails since `ssl.get_server_certificate(('localhost', 7500), ca_certs='server.pem')` works fine; the problem seems to be in the lookup/finding or linking portion of the Python/openssl relationship. – Brian M. Hunt Oct 06 '15 at 09:37
  • The big question is why Python doesn't just look the cert up on the Keychain, like it is obviously meant to. – Brian M. Hunt Oct 06 '15 at 09:39
  • 2
    For the moment I've worked around this by adding the certificate to `urlfetch_cacerts.txt` and enabled `validate_certificate`. – Brian M. Hunt Oct 06 '15 at 13:20
  • @BrianM.Hunt I've not had any luck with adding a certificate to `urlfetch_cacerts.txt`. Did you do anything other than what you described under step 2 of your original question to get this to work? I've tried adding the root CA cert, intermediate cert and the cert of the server I'm trying to talk to - without any luck. Workaround 3 didn't work for me either. :( – maltem-za May 16 '16 at 09:27
  • Sorry @bszom, that's all I've got. Another lead I just discovered though is `env_variables: GAE_USE_SOCKETS_HTTPLIB : 'anyvalue'` per https://code.google.com/p/googleappengine/issues/detail?id=12553 – Brian M. Hunt May 16 '16 at 13:45
1

I ran into the same issue on Windows. I was using an old version of Python (2.7). When I upgraded to Python 2.7.11, the problem went away.

speedplane
  • 15,673
  • 16
  • 86
  • 138
0

In my case solve this problem workaround to copy .../google-cloud-sdk/platform/lib/third_party/fancy_urllib folder to project folder.

Othervise urlfetch_stub.py in _SetupSSL cry that import fancy_urllib call ImportError: No module named fancy_urllib

SW:

  • Ubuntu 18.04
  • Python 2.7.17 / 2.7.12
  • Google Cloud SDK 319.0.0
Mintaka
  • 89
  • 1
  • 3
0

WARNING:root:/Users/<username>/Library/Python/2.7/lib/python/site-packages/lib/cacerts/urlfetch_cacerts.txt missing; without this urlfetch will not be able to validate SSL certificates.

Here's what worked for me

cd /Users/<username>/Library/Python/2.7/lib/python/site-packages/lib

Google Cloud SDK was in /Users/<username>/Documents/libraries/google-cloud-sdk/

sudo ln -s /Users/<username>/Documents/libraries/google-cloud-sdk/platform/google_appengine/lib/cacerts cacerts
Sandeep
  • 346
  • 5
  • 8