Is there a way to globally override requests' timeout setting?

Question

We have usages of the requests library littered throughout our project. Recently we came across a bug in one of our destinations where it froze mid transaction, and decided to just hold the connection open.

Naturally, our application followed suit.

Is there a environment variable, or some other way to set the timeout? Even if it's significant (say, 30 seconds) it should be enough to stop the entire works from stopping because of one service. If possible, it should be global so that I don't have to find every single use, and so that people can't forget to add it in the future.

@ShipluMokaddim that solution does not address the last sentence. — Shadow, Apr 19 '20 at 02:33
You could patch in the __init__.py of your project. Then it'll be used all the time, even for future code. I patch logging this way. — Shiplu Mokaddim, Apr 19 '20 at 06:47
@wyx , the link is now https://github.com/psf/requests/issues/3341 — Tim Tisdall, Apr 13 '22 at 13:11

score 46 · Answer 1 · answered Dec 13 '19 at 06:59

The simplest way is to "shim" the session's request function:

import requests
import functools

s = requests.Session()
s.request = functools.partial(s.request, timeout=3)

# now all get, post, head etc requests should timeout after 3 seconds
# following will fail
s.get('https://httpbin.org/delay/6')

# we can still pass higher timeout when needed
# following will succeed
s.get('https://httpbin.org/delay/6', timeout=7)

Filip Malczak · Accepted Answer · 2020-08-31T20:35:59.060

Unfortunately, looking at the code, there is no possibility to set a global default value. I was kinda surprised by that, as I would expect that to be quite common use case. If you start a feature request, please let me know (e.g. in comments to this post).

The reason for that is that methods like get(...), post(...), etc are all just thin wrappers over Session.request(...) method (requests.get(...) creates new one-shot session, just for a single request). That method takes timeout as argument, and does not inspect Session internals for a value if there is no timeout argument, so you always have to put it there manually, like 2ps proposed in his answer.

Sources:

Revised on master on 31.08.2020. Line numbers have changed, but methods stayed the same. The answer stays the same.

requests/__init__.py - import API to package scope, to provide requests.get(...)-like utilities
requests.api - API module that is imported in point above; usess one-shot sessions
requests.sessions - Session implementation
- line 337 starts Session class
- line 463 starts request(...) method
- line 526 actually uses timeout parameter
- line 534 start get(...) method

PS. See this pull request. Disclaimer: it's mine.

score 15 · Answer 3 · edited Oct 16 '19 at 11:52

15

Instead you could inherit the requests.Session class and rewrite request function, like this.

HTTP_TIMEOUT = 30

class TimeoutRequestsSession(requests.Session):
    def request(self, *args, **kwargs):
        kwargs.setdefault('timeout', HTTP_TIMEOUT)
        return super(TimeoutRequestsSession, self).request(*args, **kwargs)

session = TimeoutRequestsSession()
session.get('https://www.google.com') # connection timeout is default 30s

edited Oct 16 '19 at 11:52

Filip Malczak

3,124
24
44

answered Nov 20 '17 at 02:10

codeskyblue

408
4
6

1

This proposal would require me to find every single use, and somehow make sure people don't forget about it in the future. As such it doesn't really answer my question. – Shadow Nov 20 '17 at 02:14
2

To make it simple. You could do like this. `requests = TimeoutRequestsSession(); requests.get(...)` – codeskyblue Nov 20 '17 at 02:20
See [2ps answer](https://stackoverflow.com/a/41296110/1219585) for details how the timeout can be configured – Filip Malczak Oct 16 '19 at 11:53

score 8 · Answer 4 · answered Mar 29 '18 at 07:21

Two ways to make this happen. It involves some dirty monkey patching. Tested on python 3.6.

Put this somewhere in a file like main.py, an init.py or urls.py and make sure it is called.

Option 1: patch the requests method

import requests

def request_patch(slf, *args, **kwargs):
    print("Fix called")
    timeout = kwargs.pop('timeout', 2)
    return slf.request_orig(*args, **kwargs, timeout=timeout)

setattr(requests.sessions.Session, 'request_orig', requests.sessions.Session.request)
requests.sessions.Session.request = request_patch

Option 2: Patch the Session class

import requests

class SessionTimeoutFix(requests.Session):

    def request(self, *args, **kwargs):
        print("Fix called")
        timeout = kwargs.pop('timeout', 2)
        return super().request(*args, **kwargs, timeout=timeout)

requests.sessions.Session = SessionTimeoutFix

score 1 · Answer 5 · answered Apr 30 '19 at 06:46

1

Why not?:

    for method in ("get", "options", "head", "post", "put", "patch", "delete"):
        setattr(
            session,
            method,
            functools.partial(getattr(session, method), timeout=timeout),
        )

answered Apr 30 '19 at 06:46

vicusbass

1,714
2
19
33

2ps · Answer 6 · 2017-01-09T22:20:40.687

0

You can certainly do it per call with the timeout parameter:

requests.get('http://www.google.com', timeout=10)

The timeout parameter specifies the number of seconds or a tuple for (connection timeout, read timout). You can read more about it here.

If you want to make this global, the easiest way is to refactor all of the calls into a wrapper class and make sure that everyone is using the wrapper class to call this API endpoint. I did a quick check of the requests code and did not see the use of a global override for timeout since the default behavior is to wait for data.

edited Jan 09 '17 at 22:20

answered Dec 23 '16 at 06:14

2ps

15,099
2
27
47

nah, `counter = 1; try: request.get('http://www.google.com', timeout=5); except : counter+=1 ; if counter > My_limit : Nevermind()` – dsgdfg Dec 23 '16 at 08:35

Is there a way to globally override requests' timeout setting?

6 Answers6