Open an URL in Python

Question

How can I open an URL with a variable and key values in it in Python? I tried as below, but I got a 400 bad request. Opening the file (host) in my web browser doesn't seem to be any problem though. Perhaps the proxy?

for verb in verbs:
    host = 'http://google.no/blabla?text=' + verb + '&pos=Any'
    checktext = '<font color="maroon">' + verb + '</font>'

    params = {'http': 'http://www.someproxy.com:3128'}

    req = urllib2.Request(host, urllib.urlencode(params))
    res = urllib2.urlopen(req)

    print res.read()

NB: Got it. When I put None in as second argument in req (urllib.urlencode(params)), it works. So, it must be the proxy the server does not like.

Have a look at the [requests](http://docs.python-requests.org/en/latest/) library, which makes python HTTP vastly less troublesome. — richsilv, Jan 09 '14 at 11:08

score 0 · Answer 1 · answered Jan 09 '14 at 11:07

0

I'm not sure what you are doing with params there. The second argument to urllib2.Request is the data to POST to the server.

The way to set a proxy with Request is to call req.set_proxy(host, type) - see the documentation.

answered Jan 09 '14 at 11:07

Daniel Roseman

588,541
66
880
895

score 0 · Answer 2 · answered Jan 09 '14 at 11:13

Try using urllib2.build_opener:

import urllib2,cookielib

opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookielib.CookieJar()))
url = 'http://google.no/blabla'

for verb in verbs:
    data = '?text=' + verb + '&pos=Any' # not really sure how to fit your 'params' in here
    socket = opener.open(url,data)
    print socket.read()

Open an URL in Python

2 Answers2