Stack Overflow
Stack Overflow

Questions tagged [urllib3]

`urllib3` is a 3rd-party Python HTTP library with thread-safe connection pooling, file post, and more.

urllib3 is a library for python 2 and 3 providing a HTTP API with thread-safe connection pooling, file posting support, redirection handling and retries.

Documentation can be found at https://urllib3.readthedocs.io/en/latest/

760 questions
15
votes
2 answers

How to handle proxies in urllib3

I am having trouble finding solid examples of how to build a simple script in urllib3 which opens a url (via a proxy), then reads it and finally prints it. The proxy requires a user/pass to authenticate however it's not clear to me how you do this?…
Tom
  • 925
  • 3
  • 10
  • 24
14
votes
1 answer

What does the Python InsecureRequestWarning really mean?

I'm getting the warning: /.../local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:734: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See:…
HaPsantran
  • 5,581
  • 6
  • 24
  • 39
14
votes
2 answers

authentication with urllib3

I am trying to connect to a webpage using urllib3. The code is provided below. import urllib3 http=urllib3.PoolManager() fields={'username':'abc','password':'xyz'} r=http.request('GET',url,fields) If we assume that url is some webpage which needs…
kich
  • 734
  • 2
  • 9
  • 23
13
votes
1 answer

Change request headers between subsequent retries

Consider an http request using an OAuth token. The access token needs to be included in the header as bearer. However, if the token is expired, another request needs to be made to refresh the token and then try again. So the custom Retry object will…
SoroushA
  • 2,043
  • 1
  • 13
  • 29
12
votes
1 answer

urllib3 maxretryError

I have just started using urllib3, and I am running into a problem straightaway. According to their manuals, I started off with the simple example: Python 2.7.1+ (r271:86832, Apr 11 2011, 18:13:53) [GCC 4.5.2] on linux2 Type "help", "copyright",…
JohnJ
  • 6,736
  • 13
  • 49
  • 82
12
votes
2 answers

Script Suddenly Stops Crawling Without Error or Exception

I'm not sure why, but my script always stops crawling once it hits page 9. There are no errors, exceptions, or warnings, so I'm kind of at a loss. Can somebody help me out? P.S. Here is the full script in case anybody wants to test it for…
oldboy
  • 5,729
  • 6
  • 38
  • 86
12
votes
3 answers

How often does python-requests perform dns queries

We are using Locust to for load testing rest api services behind elastic load balancing. I came across this article regarding load balancing and auto scaling, which is something we are testing. Locust is using python-requests which is using…
djonsson
  • 603
  • 5
  • 14
12
votes
3 answers

How can I make a Post Request on Python with urllib3?

I've been trying to make a request to an API, I have to pass the following body: { "description":"Tenaris", "ticker":"TS.BA", "industry":"Metalúrgica", "currency":"ARS" } Altough the code seems to be right and it finished with "Process finished…
user1953742
  • 123
  • 1
  • 1
  • 4
12
votes
3 answers

How to perform time limited response download with python requests?

When downloading a large file with python, I want to put a time limit not only for the connection process, but also for the download. I am trying with the following python code: import requests r =…
Hristo Hristov
  • 4,021
  • 4
  • 25
  • 37
10
votes
3 answers

MaxRetryError: HTTPConnectionPool: Max retries exceeded (Caused by ProtocolError('Connection aborted.', error(111, 'Connection refused')))

I have one question:I want to test "select" and "input".can I write it like the code below: original code: 12 class Sinaselecttest(unittest.TestCase): 13 14 def setUp(self): 15 binary = FirefoxBinary('/usr/local/firefox/firefox') …
郑佳妮
  • 153
  • 1
  • 1
  • 8
10
votes
1 answer

Python 3 urllib Vs requests performance

I'm using python 3.5 and I'm checking the performance of urllib module Vs requests module. I wrote two clients in python the first one is using the urllib module and the second one is using the request module. they both generate a binary data, which…
user1470957
  • 461
  • 2
  • 7
  • 14
10
votes
1 answer

How to download a file with urllib3?

This is based on another question on this site: What's the best way to download file using urllib3 However, I cannot comment there so I ask another question: How to download a (larger) file with urllib3? I tried to use the same code that works with…
Alecz
  • 1,951
  • 1
  • 19
  • 18
10
votes
1 answer

Why does my program hang after urllib3 logs Starting new HTTPS connection?

I am trying to diagnose an issue where some of my celery worker processes appear to hang for several minutes. I have many tasks that make several IO calls (usually to third party APIs). In any given job, I might be making several thousand requests…
Scott Coates
  • 2,462
  • 5
  • 31
  • 40
9
votes
2 answers

Python - HTTP module cannot parse response if the server answers before the PUT is complete

I'm using the requests (which uses urllib3 and the Python http module under the hood) library to upload a file from a Python script. My backend starts by inspecting the headers of the request and if it doesn't comply with the needed prerequisites,…
Neumann
  • 551
  • 2
  • 6
  • 17
9
votes
3 answers

Which urllib I should choose?

as we know, python has two built-in url lib: urllib urllib2 and a third-party lib: urllib3 if my requirement is only to request a API by GET method, assume it return a JSON string. which lib I should use? do they have some duplicated…
Mathew P. Jones
  • 843
  • 1
  • 11
  • 20
1 2
3
…
50 51