python requests.get hangs until timeout for a long http request

Question

I encounter a repeating case where an http request hangs indefinitely (or until timeout) when the request is "long enough". Didn't get to test what's the threshold, but my requests are taking about 18 minutes.

I verified server responds and finishes handling the request. Also netstat verifies there is no open socket on server side.

client code:

requests.get(f"http://{ip}/data", params={...}, verify=False, headers={"User-Agent": "Python"})

It hangs on the following stack:

  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1345, in getresponse
    response.begin()
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)

Reproduces also from urllib:

>>> urllib.request.urlopen(f"http://{ip}/data").read()

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 517, in open
    response = self._open(req, data)
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 534, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 1375, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 1350, in do_open
    r = h.getresponse()
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1345, in getresponse
    response.begin()
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)

As you can see this is python 3.9.4 , although I doubt it won't reproduce for earlier releases.

I know long http requests are a bad practice, but that's not the issue I'm concerned with.

Is it your own server, or a specific existing API? If it's someone else's API, did you try checking the documentation and/or dealing with their issue tracker and/or tech support? — Karl Knechtel, Aug 05 '21 at 06:18
It is my server, and I don't think its related so server side, everything that I knew to check looks good on that end. — Mugen, Aug 05 '21 at 06:47
@Mugen I have a similar problem! A very big response causes my request to hang up and never come back (until my timeout catches the request). Did you ever find a solution for this problem? — Max, Sep 20 '22 at 15:10
@Max no I did not :( I worked hard to change the design to work with task push then poll until completed — Mugen, Sep 20 '22 at 15:14
@Mugen can you explain a bit further, what you have done? I need a workaround for this situation and can't see a possible solution. — Max, Sep 21 '22 at 05:58
If you can change the request handler, change its api to work in two steps: accept a task and return success immediately, then return status on that task - in progress, failed, completed etc. If you cannot change the other side, I'd suggest adding a proxy in a different language (where this bug don't occur) that will do the above mentioned logic, while waiting for the response in a different thread. — Mugen, Sep 21 '22 at 14:50
By the way: That's the reason why it hangs in my case: https://github.com/psf/requests/issues/5121 — Max, Oct 12 '22 at 07:18

python requests.get hangs until timeout for a long http request

0 Answers0