python ssl eof occurred in violation of protocol, wantwriteerror, zeroreturnerror

Question

I'm running many celery tasks (20,000) using gevent for the pool (also monkey patching all). Each of these tasks hit 3rd party services like adwords to pull data.

I keep having tasks fail because of underlying SSL errors. Below are the stack-traces from a few of the exceptions (in no particular order, these are failures from separate tasks). I also get WantWriteError and ZeroReturnError occasionally but the EOF error seems to come up the most.

These errors happen while using different client libraries like googleads (suds library for soap communication) as well as requests and elasticsearch. I'm guessing some of these libraries use urllib3 while others use urllib2 etc.

There has been a lot of info on the EOF issue and forcing TLSv1 but I can't seem to find a resolution that works.

I'm not sure if I'm running too many requests at once, if somethings blocking or what; any help would be greatly appreciated, I'm pulling my hair out over this one.

Traceback (most recent call last):
  ...
  File "/srv/reporting/src/reporting/stats/adwords/client.py", line 58, in _awql_report
    downloader = self._get_client(client_id).GetReportDownloader(version=self.REPORT_DOWNLOADER_VERSION)
  File "/usr/local/lib/python2.7/dist-packages/googleads/adwords.py", line 283, in GetReportDownloader
    return ReportDownloader(self, version, server)
  File "/usr/local/lib/python2.7/dist-packages/googleads/adwords.py", line 400, in __init__
    proxy=proxy_option, cache=self._adwords_client.cache).wsdl.schema
  File "/usr/local/lib/python2.7/dist-packages/suds/client.py", line 115, in __init__
    self.wsdl = reader.open(url)
  File "/usr/local/lib/python2.7/dist-packages/suds/reader.py", line 150, in open
    d = self.fn(url, self.options)
  File "/usr/local/lib/python2.7/dist-packages/suds/wsdl.py", line 136, in __init__
    d = reader.open(url)
  File "/usr/local/lib/python2.7/dist-packages/suds/reader.py", line 74, in open
    d = self.download(url)
  File "/usr/local/lib/python2.7/dist-packages/suds/reader.py", line 92, in download
    fp = self.options.transport.open(Request(url))
  File "/usr/local/lib/python2.7/dist-packages/suds/transport/https.py", line 62, in open
    return HttpTransport.open(self, request)
  File "/usr/local/lib/python2.7/dist-packages/suds/transport/http.py", line 67, in open
    return self.u2open(u2request)
  File "/usr/local/lib/python2.7/dist-packages/suds/transport/http.py", line 132, in u2open
    return url.open(u2request, timeout=tm)
  File "/usr/lib/python2.7/urllib2.py", line 400, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 418, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1216, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1178, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol>



Traceback (most recent call last):
  ...
  File "/srv/reporting/src/reporting/stats/analytics/client.py", line 57, in get_access_token
    response = requests.post('https://accounts.google.com/o/oauth2/token', data)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 88, in post
    return request('post', url, data=data, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 456, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 559, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 382, in send
    raise SSLError(e, request=request)
SSLError: [Errno bad handshake] (-1, 'Unexpected EOF')




Traceback (most recent call last):
  ...
    self.es.index(index=self.INDICE, doc_type=self.ROOT_CLASS.__name__, body=self.export(obj), id=obj.id)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py", line 68, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.py", line 213, in index
    _make_path(index, doc_type, id), params=params, body=body)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 284, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_requests.py", line 44, in perform_request
    response = self.session.request(method, url, data=body, timeout=timeout or self.timeout)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 456, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 559, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 327, in send
    timeout=timeout
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 493, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 319, in _make_request
    httplib_response = conn.getresponse(buffering=True)
  File "/usr/lib/python2.7/httplib.py", line 1030, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 407, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 365, in _read_status
    line = self.fp.readline()
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 273, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 995, in recv
    self._raise_ssl_error(self._ssl, result)
  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 851, in _raise_ssl_error
    raise ZeroReturnError()
ZeroReturnError

Answer 1

So let's break this down by each traceback block. The first ends with:

  File "/usr/lib/python2.7/urllib2.py", line 1178, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol>

This is coming from urllib2. The fact that this receives an EOF makes me think that the server closed the connection while you were waiting for that "thread" to read from the socket again. You might want to use more time.sleep(0) to yield to gevent.

The second traceback comes from requests:

  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 382, in send
    raise SSLError(e, request=request)
SSLError: [Errno bad handshake] (-1, 'Unexpected EOF')

The [Errno bad handshake] would make me tend to think this is a problem establishing the connection which could be caused by an unexpected EOF. Is that caused by using gevent? I'm uncertain.

The final traceback is definitely from requests as well but it also is coming out of PyOpenSSL and isn't being caught by urllib3 or requests.

  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 851, in _raise_ssl_error
    raise ZeroReturnError()
ZeroReturnError

I did some searching and found that "According to the pyOpenSSL docs ZeroReturnError means that the SSL connection has been closed cleanly." This says to me that the server again closed the connection because you took to long to read anything from the socket.

In short, I think you need to explicitly yield more often just to ensure that these socket problems don't arise. That's just a guess though, so take it with a grain of salt.