Google EarthEngine Python in VirtualEnv: requests can connect to google.com but ee.Initialize() fails with SSL get certificate error

Question

Trying to get a working installation of the Google EarthExplorer. I am inside a robust corporate firewall, and had to examine the certificate used by Chrome to verify the server/address being verified (using Chrome's dev tools).

After some reading - it looks like the situation is that:

1) when calling Initialize method on ee object, ee uses requests to manage the connection.

2) To configure the VirtualEnv correctly, I had to configure my virtualenv to use the organization provided certificate. Based on this SE (SE Python SSL Requests...) I was clued in to the fact that the python stack was using requests, which along with certifi manage a cert bundle for SSL on python.

3) After configuring the supplied certificate (matching that used with Chrome), I can open a connection to google inside my VirtualEnv using requests. Great!

(earthengine) X:\_01_VirtualEnvs\earthengine>python
Python 2.7.10 (default, May 23 2015, 09:40:32) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.get("https://earthengine.google.com/", verify = True)
<Response [200]>
>>>

I think this is 'working' now ... when I try the same outside of the VirtualEnv, I get failed SSL3_GET_SERVER_CERTIFICATE... message. This lets me know that I got my certificate in the right place, and it seems to be working correctly.

However, I'm still getting errors on the ee.Initialize():

>>> ee.Initialize()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "X:\_01_VirtualEnvs\earthengine\lib\site-packages\ee\__init__.py", line 9
3, in Initialize
    ApiFunction.initialize()
  File "X:\_01_VirtualEnvs\earthengine\lib\site-packages\ee\apifunction.py", lin
e 151, in initialize
    signatures = data.getAlgorithms()
  File "X:\_01_VirtualEnvs\earthengine\lib\site-packages\ee\data.py", line 410,
in getAlgorithms
    return send_('/algorithms', {}, 'GET')
  File "X:\_01_VirtualEnvs\earthengine\lib\site-packages\ee\data.py", line 738,
in send_
    response, content = send_with_backoff()
  File "X:\_01_VirtualEnvs\earthengine\lib\site-packages\ee\data.py", line 735,
in send_with_backoff
    'Unexpected HTTP error: %s' % e.message)
ee.ee_exception.EEException: Unexpected HTTP error: [Errno 1] _ssl.c:510: error:
14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

Anyone have any ideas on what is going wrong here? I thought that ee was using requests, therefore setting the python environments' requests properly.

From looking at the exception trace-back, it seems that ee has sub-class of requests based on the similarity of the trace-back signature... am I reading this right? The trace-back in both cases points towards the same _ssl.c:510 failure - failing on the same filename on the same line?

Is there a way to get more info from the exception? I'm really at a loss at this point.

score 2 · Answer 1 · answered Jun 01 '17 at 20:18

So it looks like the EE stack is using httplib2 to handle the authentication. This gives a few options:

In data.py ~ line 700 Override the SSL check (the quick and dirty):

http = httplib2.Http(timeout=(_deadline_ms / 1000.0) or None, disable_ssl_certificate_validation=True)

It looks like you should be able to explicitly direct httplib2 when instantiating the http connection object with:

HTTPLIB_CA_CERTS_PATH = os.environ.get('HTTPLIB_CA_CERTS_PATH') http = httplib2.Http(timeout=(_deadline_ms / 1000.0) or None, ca_certs=HTTPLIB_CA_CERTS_PATH)

I found this in the ca_certs_locator module, __init__.py. It is being sourced in the ee.Initialize() method (probably through something in data.py but I can't back track it. Regardless, the second option (explicitly passing the ca_certs path) doesn't solve the problem.

I'm rolling with the disable SSL validation, and using only earthengine.google.com endpoint.

score 0 · Answer 2 · answered Sep 11 '20 at 12:57

In more recent versions, the ee.Initialize() method is accepting a http_transport argument, so we no longer need to modify its source code, but rather create it in our own:

_http_transport = httplib2.Http(disable_ssl_certificate_validation=True)
ee.Initialize(credentials, http_transport=_http_transport)

This way you can also control the ca_certs option, but I haven't tried that one.

Google EarthEngine Python in VirtualEnv: requests can connect to google.com but ee.Initialize() fails with SSL get certificate error

2 Answers2