GRequests allows you to use Requests with Gevent to make asyncronous HTTP Requests easily.
GRequests allows you to use requests with gevent to make asynchronous HTTP Requests easily.
Questions tagged [grequests]
139 questions
7
votes
1 answer
Using grequests to send a pool of requests, how can I get the response time of each individual request?
I am using the grequests python library to send GET requests asynchronously to our server.
I cant figure out how to get the server response time for each individual request within the pool of requests being sent out?
unsentrequests=(grequests.get(u)…
b_dev
- 2,568
- 6
- 34
- 43
6
votes
1 answer
Is it possible to set a maximum number of retries shared between all pool connections in the same session?
Currently I do the following for setting a maximum number of connection retries for my grequest wrapper:
self._s = Session()
retries = Retry(total=5, status_forcelist=[500, 502, 503, 504])
self._s.mount('http://, HTTPAdapter(max_retries=retries))
I…
Leeren
- 1,931
- 2
- 20
- 31
6
votes
2 answers
Prevent response body download in python async http requests
I want to "ping" a server, check the header response to see if the link is broken, and if it's not broken, actually download the response body.
Traditionally, using a sync method with the requests module, you could send a get request with the stream…
undefined
- 3,949
- 4
- 26
- 38
6
votes
1 answer
How to set grequests timeout
How would I accomplish the following:
rs = (self.exporter.grequester(url) for url in url_chunk)
res_items = grequests.map(rs, timeout=10s) # this is the item that times out
David542
- 104,438
- 178
- 489
- 842
6
votes
2 answers
How to pass parameters to hooks in python grequests
According the Requests documentation, event hooks can be added to .get() function.
requests.get('http://httpbin.org', hooks=dict(response=print_url))
def print_url(r, *args, **kwargs):
print(r.url)
This is fine but how to set *args with custom…
Ved
- 3,380
- 6
- 40
- 51
5
votes
1 answer
Python: How can i send multiple HTTP requests and receive the response?
How can I send like 1000 requests the fastest way?
I know that you can send multiple request with grequests:
urls = [
'sample.url/1',
'sample.url/2',
...
]
request = (grequests.get(u) for u in urls)
print grequests.map(request)
But the…
cbl
- 211
- 1
- 3
- 6
5
votes
1 answer
Time-delayed grequests without using grequests.map()
This is the first time I've tried to use a library with less-than-ideal levels of documentation and example code, so bear with me. I have a tiny bit of experience with the Requests library, but I need to send separate requests to a specific address…
Neodied
- 119
- 1
- 8
5
votes
4 answers
Trying to install grequests on Ubuntu?
When I try to install grequests on Ubuntu with pip:
sudo pip install grequests
I get this error, but my gcc seems fine:
In file included from gevent/core.c:253:0:
gevent/libevent.h:9:19: fatal error: event.h: No such file or directory
compilation…
kfan
- 1,641
- 1
- 13
- 16
4
votes
2 answers
How to Catch a 503 Error and retry request
I'm using grequests to make about 10,000 calls, but some of these calls return as 503. This problem goes away if I don't queue all 10,000 calls at once. Breaking it into groups of 1000 seems to do the trick. However I was wondering if there's a way…
Rafael
- 3,096
- 1
- 23
- 61
4
votes
0 answers
Python: async requests with grequests not significantly faster than requests
I am just getting started in learning how to using the requests module in Python to fetch data from an API. I will be calling to this API using a very simple GET request, but I will need to do it 500,000+ times, only passing a different value for…
CurtLH
- 2,329
- 4
- 41
- 64
4
votes
2 answers
How to handle errors in GRequests?
I have this code
#!/usr/bin/python
import grequests
urls = [
'http://google.com',
'http://doesnotexists.tld'
]
def do_something(response, **kwargs):
print response.text
async_list = []
for u in urls:
action_item = grequests.get(u,…
pauts
- 119
- 1
- 8
4
votes
2 answers
gevent / requests hangs while making lots of head requests
I need to make 100k head requests, and I'm using gevent on top of requests. My code runs for a while, but then eventually hangs. I'm not sure why it's hanging, or whether it's hanging inside requests or gevent. I'm using the timeout argument inside…
vgoklani
- 10,685
- 16
- 63
- 101
3
votes
0 answers
Python: Correct way to not block a GTK UI while doing HTTP requests
Question: What is the right way to not block a GTK UI while loading data from a remote HTTP source?
Background:
I'm working on a cryptocurrency price indicator for Ubuntu Unity. I've modified the original project so that it can run multiple tickers…
bluppfisk
- 2,538
- 3
- 27
- 56
3
votes
0 answers
Beautifulsoup parse thousand pages
I have a script parsing a list with thousands of URLs.
But my problem is, that it would take ages to be done with that list.
The URL request takes around 4 seconds before the page is loaded and can be parsed.
Is there any way to parse really a large…
kratze
- 186
- 2
- 11
3
votes
0 answers
Grequests failing for a long list of urls despite size
I had a task of form:
1)send an http get to url based on a parameter
2)Modify the response based on the same parameter
3)send an http post to url based on the same parameter
I used grequests to accomplish this.Now for 55 urls it worked fine.
Then i…
vks
- 67,027
- 10
- 91
- 124