Python HTTP Error 429 with urllib2

Question

I am using the following code to resolve redirects to return a links final url

def resolve_redirects(url):
    return urllib2.urlopen(url).geturl()

Unfortunately I sometimes get HTTPError: HTTP Error 429: Too Many Requests. What is a good way to combat this? Is the following good or is there a better way.

def resolve_redirects(url):
    try:
        return urllib2.urlopen(url).geturl()
    except HTTPError:
        time.sleep(5)
        return urllib2.urlopen(url).geturl()

Also, what would happen if there is an exception in the except block?

I think that's going to depend on the URL that's returning 429 and rate limiting your calls. What's their limit/time? — c1phr, Jul 29 '15 at 17:30

score 4 · Accepted Answer · answered Jul 29 '15 at 18:14

4

It would be better to make sure the HTTP code is actually 429 before re-trying.

That can be done like this:

def resolve_redirects(url):
    try:
        return urllib2.urlopen(url).geturl()
    except HTTPError, e:
        if e.code == 429:
             time.sleep(5);
             return resolve_redirects(url)
        raise

This will also allow arbitrary numbers of retries (which may or may not be desired).

https://docs.python.org/2/howto/urllib2.html#httperror

answered Jul 29 '15 at 18:14

Jesse W at Z - Given up on SE

1,720
1
13
35

How would I specify the # of retries – Bijan Jul 29 '15 at 18:16
Yep, as I mentioned ("arbitrary number of retries"). -- a further improvement would be for it to take a param limiting the number of retries. – Jesse W at Z - Given up on SE Jul 29 '15 at 18:17

Gnarlywhale · Answer 2 · 2015-07-29T18:09:30.077

This is a fine way to handle the exception, though you should check to make sure you are always sleeping for the appropriate amount of time between requests for the given website (for example twitter limits the amount of requests per minute and has this amount clearly shown in their api documentation). So just make sure you're always sleeping long enough.

To recover from an exception within an exception, you can simply embed another try/catch block:

def resolve_redirects(url):
    try:
        return urllib2.urlopen(url).geturl()
    except HTTPError:
        time.sleep(5)
        try:
            return urllib2.urlopen(url).geturl()
        except HTTPError:
            return "Failed twice :S"

Edit: as @jesse-w-at-z points out, you should be returning an URL in the second error case, the code I posted is just a reference example of how to write a nested try/catch.

Returning a non-URL in the error case is asking for trouble. — Jesse W at Z - Given up on SE, Jul 29 '15 at 18:04
I just posted it in as a code example of a nested catch, I'll make an edit — Gnarlywhale, Jul 29 '15 at 18:06

score 0 · Answer 3 · answered Sep 14 '21 at 00:32

Adding User-Agent to request header solved my issue:

from urllib import request
from urllib.request import urlopen

url = 'https://www.example.com/abc.json'
req = request.Request(url)
req.add_header('User-Agent', 'abc-bot')
response = request.urlopen(req)

Python HTTP Error 429 with urllib2

3 Answers3