Stack Overflow
Stack Overflow

Questions tagged [urllib2]

urllib2 is a builtin python 2 module that defines functions and classes to help with URL actions. It is notably unsatisfactory and has been replaced in python 3 and by third-party libraries.

urllib2 is a module that defines functions and classes to help with URL actions (basic and digest authentication, redirections, cookies, etc). It supersedes urllib in Python 2, and in python 3 has been superseded by a new library called urllib.

In addition, the requests third party module has become a de facto standard to accomplish the same tasks.

2960 questions
83
votes
5 answers

Python urllib2 Basic Auth Problem

Update: based on Lee's comment I decided to condense my code to a really simple script and run it from the command line: import urllib2 import sys username = sys.argv[1] password = sys.argv[2] url = sys.argv[3] print("calling %s with %s:%s\n" %…
Simon
  • 1,819
  • 3
  • 18
  • 26
78
votes
7 answers

Proxy with urllib2

I open urls with: site = urllib2.urlopen('http://google.com') And what I want to do is connect the same way with a proxy I got somewhere telling me: site = urllib2.urlopen('http://google.com', proxies={'http':'127.0.0.1'}) but that didn't work…
Chris Stryker
  • 1,058
  • 2
  • 10
  • 18
74
votes
4 answers

Does python urllib2 automatically uncompress gzip data fetched from webpage?

I'm using data=urllib2.urlopen(url).read() I want to know: How can I tell if the data at a URL is gzipped? Does urllib2 automatically uncompress the data if it is gzipped? Will the data always be a string?
mlzboy
  • 14,343
  • 23
  • 76
  • 97
74
votes
4 answers

Python POST binary data

I am writing some code to interface with redmine and I need to upload some files as part of the process, but I am not sure how to do a POST request from python containing a binary file. I am trying to mimic the commands here: curl --data-binary…
Mac
  • 3,397
  • 3
  • 33
  • 58
72
votes
8 answers

Which is best in Python: urllib2, PycURL or mechanize?

Ok so I need to download some web pages using Python and did a quick investigation of my options. Included with Python: urllib - seems to me that I should use urllib2 instead. urllib has no cookie support, HTTP/FTP/local files only (no SSL) urllib2…
bigredbob
  • 1,847
  • 4
  • 19
  • 19
72
votes
3 answers

Overriding urllib2.HTTPError or urllib.error.HTTPError and reading response HTML anyway

I receive a 'HTTP Error 500: Internal Server Error' response, but I still want to read the data inside the error HTML. With Python 2.6, I normally fetch a page using: import urllib2 url = "http://google.com" data = urllib2.urlopen(url) data =…
backus
  • 4,076
  • 5
  • 28
  • 30
71
votes
6 answers

urllib2 and json

can anyone point out a tutorial that shows me how to do a POST request using urllib2 with the data being in JSON format?
pup
  • 719
  • 1
  • 5
  • 3
71
votes
8 answers

Python-Requests close http connection

I was wondering, how do you close a connection with Requests (python-requests.org)? With httplib it's HTTPConnection.close(), but how do I do the same with Requests? Code: r = requests.post("https://stream.twitter.com/1/statuses/filter.json",…
user179169
68
votes
3 answers

How do I send a custom header with urllib2 in a HTTP Request?

I want to send a custom "Accept" header in my request when using urllib2.urlopen(..). How do I do that?
Joakim
  • 11,468
  • 9
  • 44
  • 50
68
votes
2 answers

Handling urllib2's timeout? - Python

I'm using the timeout parameter within the urllib2's urlopen. urllib2.urlopen('http://www.example.org', timeout=1) How do I tell Python that if the timeout expires a custom error should be raised? Any ideas?
RadiantHex
  • 24,907
  • 47
  • 148
  • 244
65
votes
6 answers

python ignore certificate validation urllib2

I want to ignore the certification validation during my request to the server with an internal corporate link. With python requests library I would do this: r = requests.get(link, allow_redirects=False,verify=False) How do I do the same with…
Sangamesh Hs
  • 1,447
  • 3
  • 24
  • 39
61
votes
7 answers

Making a POST call instead of GET using urllib2

There's a lot of stuff out there on urllib2 and POST calls, but I'm stuck on a problem. I'm trying to do a simple POST call to a service: url = 'http://myserver/post_service' data = urllib.urlencode({'name' : 'joe', 'age' :…
alfonso.kim
  • 2,844
  • 4
  • 32
  • 37
59
votes
7 answers

Python Requests: Post JSON and file in single request

I need to do a API call to upload a file along with a JSON string with details about the file. I am trying to use the python requests lib to do this: import requests info = { 'var1' : 'this', 'var2' : 'that', } data = json.dumps({ …
oznu
  • 1,604
  • 2
  • 12
  • 11
57
votes
6 answers

Python's `urllib2`: Why do I get error 403 when I `urlopen` a Wikipedia page?

I have a strange bug when trying to urlopen a certain page from Wikipedia. This is the page: http://en.wikipedia.org/wiki/OpenCola_(drink) This is the shell session: >>> f = urllib2.urlopen('http://en.wikipedia.org/wiki/OpenCola_(drink)') Traceback…
Ram Rachum
  • 84,019
  • 84
  • 236
  • 374
56
votes
4 answers

Stream large binary files with urllib2 to file

I use the following code to stream large files from the Internet into a local file: fp = open(file, 'wb') req = urllib2.urlopen(url) for line in req: fp.write(line) fp.close() This works but it downloads quite slowly. Is there a faster way?…
hoju
  • 28,392
  • 37
  • 134
  • 178
1
2
3
…
99 100