How to check redirected web page address, without downloading it in Python

Question

For a given url, how can I detect final internet location after HTTP redirects, without downloading final page (e.g. HEAD request.) using python. I am trying to write a mass downloader, my downloading mechanism needs to know internet location of page before downloading it.

edit

I ended up doing this, I hope this helps other people. I am still open to other methods.

import urlparse
import httplib

def getFinalUrl(url):
    "Navigates Through redirections to get final url."
    parsed = urlparse.urlparse(url)
    conn = httplib.HTTPConnection(parsed.netloc)
    conn.request("HEAD",parsed.path)
    response = conn.getresponse()
    if str(response.status).startswith("3"):
        new_location = [v for k,v in response.getheaders() if k == "location"][0]
        return getFinalUrl(new_location)
    return url

score 1 · Answer 1 · answered Sep 19 '12 at 10:04

I strongly suggest you to use requests library. It is well coded and actively maintained. Requests can make anything you need like prefetch/

From the Requests' documentation http://docs.python-requests.org/en/latest/user/advanced/ :

By default, when you make a request, the body of the response is downloaded immediately. You can override this behavior and defer downloading the response body until you access the Response.content attribute with the prefetch parameter:

tarball_url = 'https://github.com/kennethreitz/requests/tarball/master'
r = requests.get(tarball_url, prefetch=False)

At this point only the response headers have been downloaded and the connection remains open, hence allowing us to make content retrieval conditional:

if int(r.headers['content-length']) < TOO_LONG:
  content = r.content
  ...

You can further control the workflow by use of the Response.iter_content and Response.iter_lines methods, or reading from the underlying urllib3 urllib3.HTTPResponse at Response.raw

This now is `r = requests.get(tarball_url, stream=True)` – dirkbo Sep 02 '20 at 18:00 — dirkbo, Sep 02 '20 at 18:00

score 0 · Answer 2 · answered Sep 20 '11 at 11:42

0

You can use httplib to send HEAD requests.

answered Sep 20 '11 at 11:42

Amber

507,862
82
626
550

How about getting thorugh redirections. Should I need to do that manually? – yasar Sep 20 '11 at 11:46

score 0 · Answer 3 · answered Sep 20 '11 at 12:08

0

You can also have a look at python-requests, which seems to be the new trendy API for HTTP requests, replacing the possibly awkward httplib2. (see Why Not httplib2)

It also has a head() method for this.

answered Sep 20 '11 at 12:08

tsimbalar

5,790
6
37
61

How to check redirected web page address, without downloading it in Python

edit

3 Answers3

Linked