How to get log of every retry attempt made by Python Requests Library?

Question

I'm using following code to implement retry mechanism in python requests.

from requests import Session
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

s = Session()
retries = Retry(total=5,
                raise_on_redirect=True, 
                raise_on_status=True,
                backoff_factor=1,
                status_forcelist=[ 500, 502, 503, 504 ])
s.mount('http://', HTTPAdapter(max_retries=retries))
s.mount('https://', HTTPAdapter(max_retries=retries))
response=s.get('https://example.com')

It is working perfectly, however all this happens silently. What I want is to get informed whenever a retry attempt is made and if possible why that retry was happened. I want something like this for every retry attempt.

print(Exception.__class__)
print('Retrying after' + x + 'seconds')

Is it possible?

score 10 · Accepted Answer · answered Oct 27 '21 at 13:51

10

You can write your own Class to add logs it should be inherited from the Retry.

class LogRetry(Retry):
  """
     Adding extra logs before making a retry request     
  """      
  def __init__(self, *args, **kwargs):
    logger.info(f'<add your logs here>')
    super().__init__(*args, **kwargs)

retries = LogRetry(<your args>)

answered Oct 27 '21 at 13:51

ramganesh

741
1
8
33

Doesn't this just log with the retry object is created? It doesn't actually log the retry. – Dan Diephouse Jan 20 '23 at 18:01
@DanDiephouse urllib3 creates new instances of the respective Retry class with the new number of retries left if a retry happens (it's implemented in a recursive fashion) https://github.com/urllib3/urllib3/blob/96588e9bf66be564c7437da8b1c2c054bc3142b4/src/urllib3/util/retry.py – Andre Wisplinghoff Jan 25 '23 at 17:54

score 1 · Answer 2 · answered Jul 16 '23 at 16:40

Extremely hacky but it works, it won't log when a retry isn't issued.

class LogOnRetry(Retry):
  
    def __init__(self, *args, __logger: logging.Logger = None,**kwargs):
        self.__logger = kwargs.pop("__logger")
        super().__init__(*args, **kwargs)

    def new(self, **kw) -> Retry:
        kw["__logger"] = self.__logger
        return super().new(**kw)

    def increment(self, method: str | None = None, url: str | None = None, response: BaseHTTPResponse | None = None, error: Exception | None = None, _pool: ConnectionPool | None = None, _stacktrace: TracebackType | None = None) -> Retry:
        if self.__logger:
            pass
            # log here, get access to url, method and response used.
        return super().increment(method, url, response, error, _pool, _stacktrace)

score 0 · Answer 3 · answered Jan 20 '23 at 18:03

Best option I found was to increase logging for urllib:

    'urllib3.util.retry':
    level: DEBUG
    handlers: [console]
    propagate: no

Output is:

2023-01-20 09:

59:41.018 | DEBUG    | logging:callHandlers:1706 - Starting new HTTP connection (1): x.x.x.x:8001
2023-01-20 09:59:48.263 | DEBUG    | logging:callHandlers:1706 - Incremented Retry for (url='/path/'): LogRetry(total=4, connect=None, read=None, redirect=None, status=None)
2023-01-20 09:59:48.264 | WARNING  | logging:callHandlers:1706 - Retrying (LogRetry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x28894bf10>, 'Connection to x.x.x.x timed out. (connect timeout=5)')': /path/

How to get log of every retry attempt made by Python Requests Library?

3 Answers3

Linked