Stack Overflow
Stack Overflow

Questions tagged [urllib]

Python module providing a high-level interface for fetching data across the World Wide Web. Predecessor to urllib2. In Python 3, urllib2 and urllib have been reorganized and merged into urllib.

For Python 2, the urllib module is the predecessor of the urllib2 module, while the latter still uses some functionality of the former.

For Python 3, the urllib package was reorganized. It now has no content of its own. All methods and classes are in several submodules:

Note that doesn't exist in Python 3 anymore.

3960 questions
1
vote
1 answer

Python: urllib2 handle multiple openers

I have to keep 2 urllib2 openers, one for direct requests and the second to make requests via proxy server and I've to rebuild opener switch between requests. How to keep context openers for example direct and proxy separately?
sultan
  • 5,978
  • 14
  • 59
  • 103
1
vote
1 answer

python - urllib.request.urlretrieve throws unexpected exception unknown url type: ' '

I am trying to download files using urllib.request.retrieve() I am using Python 3 and the downloads are successful, but I don't know why it throws exception. For some reason it throws an exception. This is the main file: import os import…
Aniket Chowdhury
  • 332
  • 3
  • 13
1
vote
2 answers

Log in a site and navigate another pages

I have a script for Python 2 to login into a webpage and then move inside to reach a couple of files pointed to on the same site, but different pages. Python 2 let me open the site with my credentials and then create a opener.open() to keep the…
Andres Urrego Angel
  • 1,842
  • 7
  • 29
  • 55
1
vote
1 answer

Get and Put requests for json data format using urllib(2)

I need to implement "put" and "get" REST API requests for sending a JSON file. The problem is that it has to be done using urllib or urllib2 module (e.g. no requests module). Is there any brief tutorial on how to do it? Thanks!
Nastya
  • 21
  • 6
1
vote
1 answer

Python - Issue Scraping with BeautifulSoup

I'm trying to scrape the Stack Overflow jobs page using Beautiful Soup 4 and URLLIB as a personal project. I'm facing an issue where I'm trying to scrape all the links to the 50 jobs listed on each page. I'm using a regex to identify these links.…
joshuaneeraj
  • 13
  • 3
1
vote
2 answers

Accessing Elasticsearch with Python 3

I want to use the Python 3 module urllib to access an Elasticsearch database at localhost:9200. My script gets a valid request (generated by Kibana) piped to STDIN in JSON format. Here is what I did: import json import sys import urllib.parse import…
eins6180
  • 163
  • 5
1
vote
1 answer

Scraping multiple web pages with BeautifulSoup when the first page link is different from others

I'm trying to scrape this page! for a project i'm working on. I want to get the details (Price, Mileage, Transmission, and Age) of each car for all the pages. The issues I'm having with my code below are: The first page link is different from…
sudoh
  • 93
  • 1
  • 7
1
vote
2 answers

python3 urlopen raises 404 error while openning in the browser

I'm using urllib.request with python 3.4.6 to open https://www.ethz.ch/ (The actual url is longer but the problem is the same), which opens fine with Firefox but throws a 404 error with python. Here is the code from urllib.request import…
Pheidippides
  • 83
  • 6
1
vote
1 answer

Download Progress bar in Kivy

I am trying to build an app that downloads a file, whose progress can be tracked on a kivy app. I have looked at the example here and here for the download progress. This is my code: from kivy.app import App from kivy.uix.button import Button from…
Echchama Nayak
  • 971
  • 3
  • 23
  • 44
1
vote
1 answer

Google App Engine Ubuntu 14.04 urlfetch 500 / 200 issue (Python 2.7)

I hope this saves somebody some time. Posting because I found very little concerning URLFetch error. I was suddenly receiving "WARNING 2017-06-28 23:09:40,971 urlfetch_stub.py:550] Stripped prohibited headers from URLFetch request: ['Host']" on a…
Steve Kritzer
  • 29
  • 4
1
vote
1 answer

poloniex 403 forbidden using python3.5

I am trying to create a basic authenticated connection to poloniex and I keep getting a 403 forbidden error returned back from their API. time import time import urllib.request import urllib.parse import hashlib import hmac APIkey =…
nix
  • 183
  • 2
  • 7
1
vote
0 answers

How to download all links from a ftp link?

Is there a way to download all files from a ftp link, preferably not overwriting if same file already exists in download folder? I can download specific links like this: import urllib urllib.urlretrieve('ftp://server/path/to/file', 'file')
user308827
  • 21,227
  • 87
  • 254
  • 417
1
vote
1 answer

urllib.urlretrieve downloads empty zip files

I'm trying to download a export of space in a zip file. But somehow python downloads a empty and corrupted zip file. When you download the file manual by the browser everything is ok. I use Python 2.7.13 #!/usr/bin/python import xmlrpclib import…
Danny
  • 189
  • 2
  • 15
1
vote
1 answer

Unable to decode unicode for Stack Exchange API

I was looking at this codegolf problem, and decided to try taking the python solution and use urllib instead. I modified some sample code for manipulating json with urllib: import urllib.request import json res =…
Dair
  • 15,910
  • 9
  • 62
  • 107
1
vote
0 answers

What is the correct way to add headers (user-agents) when scraping with urllib & Python 3?

I am trying to add a user agent to the scrape I'm doing with urllib and BeautifulSoup in Python 3. Here's my code import bs4 as bs import urllib.request import urllib.parse from random import choice from time import sleep import os user_agents = [ …
pocallaghan88
  • 23
  • 1
  • 6
1 2 3
…
99
100