Python problems with FancyURLopener, 401, and "Connection: close"

Question

I'm new to Python, so forgive me if I am missing something obvious.

I am using urllib.FancyURLopener to retrieve a web document. It works fine when authentication is disabled on the web server, but fails when authentication is enabled.

My guess is that I need to subclass urllib.FancyURLopener to override the get_user_passwd() and/or prompt_user_passwd() methods. So I did:

class my_opener (urllib.FancyURLopener):

    # Redefine
    def get_user_passwd(self, host, realm, clear_cache=0):
        print "get_user_passwd() called; host %s, realm %s" % (host, realm)
        return ('name', 'password')

Then I attempt to open the page:

try:
    opener = my_opener()
    f = opener.open ('http://1.2.3.4/whatever.html')
    content = f.read()
    print "Got it:  ", content

except IOError:
    print "Failed!"

I expect FancyURLopener to handle the 401, call my get_user_passwd(), and retry the request.

It does not; I get the IOError exception when I call "f = opener.open()".

Wireshark tells me that the request is sent, and that the server is sending a "401 Unauthorized" response with two headers of interest:

WWW-Authenticate: BASIC
Connection: close

The connection is then closed, I catch my exception, and it's all over.

It fails the same way even if I retry the "f = opener.open()" after IOError.

I have verified that my my_opener() class is working by overriding the http_error_401() method with a simple "print 'Got 401 error'". I have also tried to override the prompt_user_passwd() method, but that doesn't happen either.

I see no way to proactively specify the user name and password.

So how do I get urllib to retry the request?

Thanks.

Just tried using urllib2.HTTPBasicAuthHandler() with the add_password() method. In this case, the server does not return 401 Unauthorized; instead it returns 200 OK with an error message. The difference: with urllib2, the request includes the "Connection: close" header. The urllib.FancyURLopener does not include this header. — Martin Del Vecchio, Mar 04 '10 at 20:48

score 0 · Answer 1 · answered Jul 15 '11 at 21:08

I just tried your code on my webserver (nginx) and it works as expected:

Get from urllib client

HTTP/1.1 401 Unauthorized from server with Headers

Connection: close
WWW-Authenticate: Basic realm="Restricted"

client tries again with Authorization header

Authorization: Basic <Base64encoded credentials>

Server responds with 200 OK + Content

So I guess your code is right (I tried it with python 2.7.1) and maybe the webserver you are trying to access is not working as expected. Here is the code tested using the free http basic auth testsite browserspy.dk (seems they are using apache - the code works as expected):

import urllib

class my_opener (urllib.FancyURLopener):

    # Redefine
    def get_user_passwd(self, host, realm, clear_cache=0):
        print "get_user_passwd() called; host %s, realm %s" % (host, realm)
        return ('test', 'test')

try:
    opener = my_opener()
    f = opener.open ('http://browserspy.dk/password-ok.php')
    content = f.read()
    print "Got it:  ", content

except IOError:
    print "Failed!"

Python problems with FancyURLopener, 401, and "Connection: close"

1 Answers1