Stack Overflow
Stack Overflow

Questions tagged [urllib2]

urllib2 is a builtin python 2 module that defines functions and classes to help with URL actions. It is notably unsatisfactory and has been replaced in python 3 and by third-party libraries.

urllib2 is a module that defines functions and classes to help with URL actions (basic and digest authentication, redirections, cookies, etc). It supersedes urllib in Python 2, and in python 3 has been superseded by a new library called urllib.

In addition, the requests third party module has become a de facto standard to accomplish the same tasks.

2960 questions
55
votes
10 answers

How to fetch a non-ascii url with urlopen?

I need to fetch data from a URL with non-ascii characters but urllib2.urlopen refuses to open the resource and raises: UnicodeEncodeError: 'ascii' codec can't encode character u'\u0131' in position 26: ordinal not in range(128) I know the URL is…
onurmatik
  • 5,105
  • 7
  • 42
  • 67
54
votes
3 answers

Python 3 urllib produces TypeError: POST data should be bytes or an iterable of bytes. It cannot be of type str

I am trying to convert working Python 2.7 code into Python 3 code and I am receiving a type error from the urllib request module. I used the inbuilt 2to3 Python tool to convert the below working urllib and urllib2 Python 2.7 code: import…
Greg
  • 8,175
  • 16
  • 72
  • 125
52
votes
8 answers

Python: urllib/urllib2/httplib confusion

I'm trying to test the functionality of a web app by scripting a login sequence in Python, but I'm having some troubles. Here's what I need to do: Do a POST with a few parameters and headers. Follow a redirect Retrieve the HTML body. Now, I'm…
Ace
  • 4,443
  • 6
  • 38
  • 46
51
votes
6 answers

Using MultipartPostHandler to POST form-data with Python

Problem: When POSTing data with Python's urllib2, all data is URL encoded and sent as Content-Type: application/x-www-form-urlencoded. When uploading files, the Content-Type should instead be set to multipart/form-data and the contents be…
Dan
  • 1,721
  • 3
  • 16
  • 20
51
votes
5 answers

Using an HTTP PROXY - Python

I familiar with the fact that I should set the HTTP_RPOXY environment variable to the proxy address. Generally urllib works fine, the problem is dealing with urllib2. >>> urllib2.urlopen("http://www.google.com").read() returns urllib2.URLError:…
RadiantHex
  • 24,907
  • 47
  • 148
  • 244
51
votes
5 answers

How to make HTTP DELETE method using urllib2?

Does urllib2 support DELETE or PUT method? If yes provide with any example please. I need to use piston API.
Pol
  • 24,517
  • 28
  • 74
  • 95
48
votes
3 answers

How can I use a SOCKS 4/5 proxy with urllib2?

How can I use a SOCKS 4/5 proxy with urllib2 to download a web page?
Mike
  • 483
  • 1
  • 5
  • 4
47
votes
4 answers

How do I prevent Python's urllib(2) from following a redirect

I am currently trying to log into a site using Python however the site seems to be sending a cookie and a redirect statement on the same page. Python seems to be following that redirect thus preventing me from reading the cookie send by the login…
Jack Edmonds
  • 31,931
  • 18
  • 65
  • 77
46
votes
2 answers

urllib2 read to Unicode

I need to store the content of a site that can be in any language. And I need to be able to search the content for a Unicode string. I have tried something like: import urllib2 req = urllib2.urlopen('http://lenta.ru') content = req.read() The…
Vitaly Babiy
  • 6,114
  • 4
  • 26
  • 24
42
votes
3 answers

Python and urllib2: how to make a GET request with parameters

I'm building an "API API", it's basically a wrapper for a in house REST web service that the web app will be making a lot of requests to. Some of the web service calls need to be GET rather than post, but passing parameters. Is there a "best…
adamJLev
  • 13,713
  • 11
  • 60
  • 65
42
votes
3 answers

urllib.quote() throws KeyError

To encode the URI, I used urllib.quote("schönefeld") but when some non-ascii characters exists in string, it thorws KeyError: u'\xe9' Code: return ''.join(map(quoter, s)) My input strings are köln, brønshøj, schönefeld etc. When I tried just…
Garfield
  • 2,487
  • 4
  • 31
  • 54
42
votes
7 answers

Python urllib2 with keep alive

How can I make a "keep alive" HTTP request using Python's urllib2?
ibz
  • 44,461
  • 24
  • 70
  • 86
41
votes
4 answers

Download and decompress gzipped file in memory?

I would like to download a file using urllib and decompress the file in memory before saving. This is what I have right now: response = urllib2.urlopen(baseURL + filename) compressedFile =…
OregonTrail
  • 8,594
  • 7
  • 43
  • 58
40
votes
5 answers

How to send a POST request using django?

I dont want to use html file, but only with django I have to make POST request. Just like urllib2 sends a get request.
zjm1126
  • 34,604
  • 53
  • 121
  • 166
40
votes
10 answers

Fetch a Wikipedia article with Python

I try to fetch a Wikipedia article with Python's urllib: f = urllib.urlopen("http://en.wikipedia.org/w/index.php?title=Albert_Einstein&printable=yes") s = f.read() f.close() However instead of the html page I get the following response:…
dkp
  • 825
  • 3
  • 9
  • 14
1 2
3
…
99 100