Python requests - retry for any status code but not 400

Question

I am using the following snippet(please note, only partial snippets shown):

from urllib3.util import Retry
status_forcelist = (500, 502, 504)


retry = Retry(
        total=retries,
        read=retries,
        connect=retries,
        backoff_factor=backoff_factor,
        status_forcelist=status_forcelist,
        method_whitelist=frozenset(['GET', 'POST'])

As of now, the retry would be done for status codes 500,502,504. However, I intend to retry for any status code but not 400. Is there a graceful way to achieve this instead of populating/hardcoding the status_forcelist for all status codes ?

Patrick Artner · Accepted Answer · 2018-08-06T06:00:31.017

10

You can take the codes from request module - which saves you writing them out. In case any new ones get introduced you would have them as well:

import requests
from urllib3.util import Retry

status_forcelist = tuple( x for x in requests.status_codes._codes if x != 400)

print(status_forcelist)

Output:

(100, 101, 102, 103, 122, 200, 201, 202, 203, 204, 205, 206, 207, 208, 226, 300, 301, 302, 
 303, 304, 305, 306, 307, 308, 401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 
 413, 414, 415, 416, 417, 418, 421, 422, 423, 424, 425, 426, 428, 429, 431, 444, 449, 450, 
 451, 499, 500, 501, 502, 503, 504, 505, 506, 507, 509, 510, 511)

Caveat: wikipedia list "self created" / unofficial statuscodes as well as proprietary ones around 9xx which are not covered by this list though. @zwer solution would cover those as well.

edited Aug 06 '18 at 06:00

answered Aug 06 '18 at 05:55

Patrick Artner

50,409
9
43
69

Why isn't `104 (Connection reset by peer)` in this list? – Amir Shabani Apr 02 '20 at 08:30
1

@amirA sorry for the late answer, your question escaped me. It is not in it because at the time of answering 104 was not a thing inside `requests.status_codes` in the version `requests` I used. It will only ever reflect the ones that are in it - see the added _Caveat_ remark 2years before your question. – Patrick Artner Jan 30 '21 at 10:10

score 6 · Answer 2 · answered Aug 06 '18 at 05:34

status_forcelist, unfortunately, applies as a whitelist only (given the use, it can be called a blacklist, tho) so there are no built-in ways to do it gracefully. However, you can override (or monkey-patch if that's your thing) urllib3.util.retry.Retry.is_retry() to treat it as a blacklist, something like:

import urllib3.util.retry

class BlacklistRetry(urllib3.util.retry.Retry):

    def is_retry(self, method, status_code, has_retry_after=False):
        if not self._is_method_retryable(method):
            return False

        if self.status_forcelist and status_code not in self.status_forcelist:
            # threat as a blacklist --------------^
            return True

        return (self.total and self.respect_retry_after_header and
                has_retry_after and (status_code in self.RETRY_AFTER_STATUS_CODES))

Then use it instead of the original Retry and just add 400 to its status_forcelist.

score 1 · Answer 3 · answered Sep 04 '22 at 19:29

The current accepted answer has a bug, it retries for successful http status codes, which means it will unnecessarily retry - if you want proof, you can test this with the code from this answer here.

You could just follow a similar implementation to Response.raise_for_status.

from urllib3.util import Retry
retry = Retry(
        total=retries,
        read=retries,
        connect=retries,
        backoff_factor=backoff_factor,
        status_forcelist=tuple(range(401, 600)),
        method_whitelist=frozenset(['GET', 'POST'])

Python requests - retry for any status code but not 400

3 Answers3