Python: requests.exceptions.ConnectionError - Only one usage of each socket address (protocol/network address/port) is normally permitted

Question

I'm sending GET requests to a json API inside a loop, like this:

import requests

for var in my_range:
    url = f"http://127.0.0.1:8081/api/my_query/{var}"
        response = requests.get(url=url)
        if response.status_code == 200:
            < do stuff >

I'm able to send ~ 236 requests / s, and my script runs for a few minutes, however I think all of my sockets are binding up before the TIME_WAIT state expires and they become available for reuse.

requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8081): Max retries exceeded with url: /api/my_query/{iter} (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x03E4BCD0>: Failed to establish a new connection: [WinError 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted'))

I see that I can lower the TIME_WAIT state from 240 s to 30 s, however apparently that can cause other undesirable issues. I also see an answer calling on SO_REUSEADDR from the sockets module but I couldn't find anything about using that solution in conjunction with the requests module. I tried response.close() after each request in the loop but the error persists.

Wondering how to modify my code to solve this issue?

https://requests.readthedocs.io/en/master/user/advanced/ So if you’re making several requests to the same host, the underlying TCP connection will be reused, which can result in a significant performance increase (see HTTP persistent connection). Not sure if this helps. — James Lin, Oct 07 '20 at 03:24
Sessions are usually easy to use. Just wrap `with requests.Session() as s:` around your for loop, and then replace `requests.get` with `s.get`. — Frank Yellin, Oct 07 '20 at 03:33

McQuestion · Accepted Answer · 2020-12-19T18:45:17.877

0

Thanks to James Lin and Frank Yellin for the tip!

import requests

with requests.Session() as s:
    for var in my_range:
        url = f"http://127.0.0.1:8081/api/my_query/{var}"
            response = s.get(url=url)
            if response.status_code == 200:
                < do stuff >

Problem solved

edited Dec 19 '20 at 18:45

answered Oct 07 '20 at 05:08

McQuestion

51
1
9

Python: requests.exceptions.ConnectionError - Only one usage of each socket address (protocol/network address/port) is normally permitted

1 Answers1

Linked