0

I am trying to use the urllib3 Retry functionality when scraping a website, but have found that it's not very verbose in the way it lets you know when it's retrying.

My class looks like this:

class Session:
    def __init__(self):
        self.session = requests.session()
        retry = Retry(total=10, backoff_factor=1, status_forcelist=[413,429,500,502,503,504])
        adapter = HTTPAdapter(max_retries=retry)
        self.session.mount("http://", adapter)
        self.session.mount("https://", adapter)

    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        self.session.__exit__()

    def get(self, url, params={}, headers=None):
        """
        Implements the get function defined below
        """
        return get(url, self.session, params, headers)


def get(url: str, session=None, params=None, headers=None,) -> requests.Response:
    """
    Function to return the response from a request given certain parameters, including auth and session
    """
    logger.info("Get request to url: %s", url)

    if session:
        response = session.get(url, params=params, headers=headers,)
    else:
        response = requests.get(url, params=params, headers=headers,)
    response.raise_for_status()

    return response

I want to be able to add some logging, such as:

failed on 1st attempt, backing off for 1s...
failed on 2nd attempt, backing off for 2s...

etc. Is there any way to do this with the Retry functionality other than writing something bespoke?

jonrsharpe
  • 115,751
  • 26
  • 228
  • 437
Jack Ballinger
  • 11
  • 1
  • Have you tried [adjusting the log level](https://urllib3.readthedocs.io/en/stable/user-guide.html?highlight=log#logging) on urllib3 generally? – jonrsharpe Feb 16 '22 at 16:56
  • this is a good shout, although it only returns info on the current retry number, rather than the timeout - presumably that means there isn't anything that can be done considering the information doesn't seem to be there at the lowest level of logging? – Jack Ballinger Feb 16 '22 at 17:09
  • you can try [get-log-of-every-attempt](https://stackoverflow.com/a/69739940/4628154), and add more custom details you want. – ramganesh Nov 26 '22 at 23:11

0 Answers0