httplib is not getting all the redirect codes

Question

I am trying to get the final url of a page that seems to redirect more than once. Try this sample URL in your browser and compare it to the final URL at the bottom of my code snippet:

Link that redirects more than once

And here is the test code I was running, notice the final URL that gets a code of 200 isn't the same as the one in your browser. What are my options?

Python 2.7.1+ (r271:86832, Apr 11 2011, 18:13:53)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import httplib
>>> from urlparse import urlparse
>>> url = 'http://www.usmc.mil/units/hqmc/'
>>> host = urlparse(url)[1]
>>> req = ''.join(urlparse(url)[2:5])
>>> conn = httplib.HTTPConnection(host)
>>> conn.request('HEAD', req)
>>> resp = conn.getresponse()
>>> print resp.status
    301
>>> print resp.msg.dict['location']
    http://www.marines.mil/units/hqmc/

>>> url = 'http://www.marines.mil/units/hqmc/'
>>> host = urlparse(url)[1]
>>> req = ''.join(urlparse(url)[2:5])
>>> conn = httplib.HTTPConnection(host)
>>> conn.request('HEAD', req)
>>> resp = conn.getresponse()
>>> print resp.status
    302
>>> print resp.msg.dict['location']
    http://www.marines.mil/units/hqmc/default.aspx

>>> url = 'http://www.marines.mil/units/hqmc/default.aspx'
>>> host = urlparse(url)[1]
>>> req = ''.join(urlparse(url)[2:5])
>>> conn = httplib.HTTPConnection(host)
>>> conn.request('HEAD', req)
>>> resp = conn.getresponse()
>>> print resp.status
    200
>>> print resp.msg.dict['location']
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    KeyError: 'location'
>>> print url
    http://www.marines.mil/units/hqmc/default.aspx //THIS URL DOES NOT RETURN A 200 IN ANY BROWSER I HAVE TRIED

score 5 · Answer 1 · answered Jul 23 '12 at 17:54

You can use HttpLib2 to get the actual location of an URL:

import httplib2

def getContentLocation(link):
    h = httplib2.Http(".cache_httplib")
    h.follow_all_redirects = True
    resp = h.request(link, "GET")[0]
    contentLocation = resp['content-location']
    return contentLocation

if __name__ == '__main__':
    link = 'http://podcast.at/podcast_url344476.html'
    print getContentLocation(link)

Execution looks like this:

$ python2.7 getContentLocation.py
http://keyinvest.podcaster.de/8uhr30.rss

Note this example also uses caching (which is not supported neither by urllib nor by httplib). So this will run repeatedly significantly faster. This might be interesting for crawling/scraping. If you do not want caching, replace h = httplib2.Http(".cache_httplib") with h = httplib2.Http().

ihucos · Accepted Answer · 2011-05-28T00:50:56.260

3

you can try to set your User-Agent header to the User-Agent of your browser.

ps: urllib2 automatically redirects

EDIT:

In [2]: import urllib2
In [3]: resp = urllib2.urlopen('http://www.usmc.mil/units/hqmc/')
In [4]: resp.geturl()
Out[4]: 'http://www.marines.mil/units/hqmc/default.aspx

edited May 28 '11 at 00:50

answered May 28 '11 at 00:39

ihucos

1,821
3
19
30

1

I know, but urllib2 does not tell me what the final proper url is. That's what I am trying to find. – Nathan May 28 '11 at 00:41
Open that final URL you got `http://www.marines.mil/units/hqmc/default.aspx` in your browser. It redirects to `http://www.marines.mil/unit/hqmc/Pages/default.aspx`. I checked the server response code in chrome and firefox. Those browsers detect a 302 redirect when python says 200 – Nathan May 28 '11 at 00:53
maybe there is a redirect by javascript or html? – ihucos May 28 '11 at 01:00
4

jep, `` – ihucos May 28 '11 at 01:02
Nice catch. Just saw that too. I will find a good link on capturing meta refresh redirects, or even javascript ones. Thanks! – Nathan May 28 '11 at 01:04

httplib is not getting all the redirect codes

2 Answers2

Linked