GRequests allows you to use Requests with Gevent to make asyncronous HTTP Requests easily.
GRequests allows you to use requests with gevent to make asynchronous HTTP Requests easily.
Questions tagged [grequests]
139 questions
1
vote
2 answers
How do I use Grequests on AWS Lambda?
I have a Lambda function in Python that uses several libraries with binary dependencies: numpy, scipy, Pillow, etc. In the past, I have successfully compiled them on Amazon Linux & deployed to Lambda.
I recently added Grequest to the pile of…
Brandon
- 7,736
- 9
- 47
- 72
1
vote
1 answer
requests + grequests: is the "Connection pool is full, discarding connection:" warning relevant?
I'm hosting a server on localhost and I want to fire hundreds of GET requests asynchronously. For this I am using grequests. Everything appears to work fine but I repeatedly get the…
roganjosh
- 12,594
- 4
- 29
- 46
1
vote
2 answers
POSTing and GETing with grequests
I'm using grequests to scape websites faster. However, I also need to login to the website.
Before (just using requests) I could do:
where headers is my User-Agent.
with requests.Session() as s:
s.headers.update(headers)
s.post(loginURL,…
Rafael
- 3,096
- 1
- 23
- 61
1
vote
1 answer
sending more than1000 requests per second in python
I am trying to send requests concurrently to a server and then record the average latency using this code:
import Queue
import time
import threading
import urllib2
data = "{"image_1":"abc/xyz.jpg"}"
headers = {.....}
def get_url(q, url):
num =…
the_unknown_spirit
- 2,518
- 7
- 34
- 56
1
vote
0 answers
parsing: grequests "None" responses: connection closed by host?
I am trying out some code which works great when I only have ~50 requests to send, but now I'm trying with more requests (about 200) and I feel like the server might be closing the connection after 60 requests, because I get "None" responses.
Here's…
vandernath
- 3,665
- 3
- 15
- 24
1
vote
1 answer
Completely disable grequests logging
How would I suppress the following debugging error messages I'm getting from grequests?
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gevent/greenlet.py", line 327, in run
result = self._run(*self.args,…
David542
- 104,438
- 178
- 489
- 842
1
vote
0 answers
'Address family not supported by protocol' errors when using grequests
When I invoke the following function to process a long list of URLs (accessing the same site (i.e. http://foo.bar.com/url1, http://foo.bar.com/url2 etc):
import time
import grequests
def processUrls(block=2500, write=100000, timeout=0.5):
urls…
I Z
- 5,719
- 19
- 53
- 100
1
vote
1 answer
How to make Asynchronous http request using python grequests
I am not able to get the asynchronous property out of this code. The calling url has a sleep of 10 seconds.
The grequests.send command finishes immediately but I am not getting the request at the server.
If I am using r.send() or…
Carbonrock
- 457
- 2
- 15
1
vote
1 answer
Grequests with exponential backoff
The following is a pattern I have for doing simultaneous requests:
rs = (grequests.get(url) for url in urls)
res_items = grequests.map(rs)
for num, res in enumerate(res_items):
json_data = json.loads(res.text)
However, this crashes with the…
David542
- 104,438
- 178
- 489
- 842
1
vote
2 answers
POSTing json data with grequests?
I need to make async POST requests with grequests.
My post body (in json), is this:
[{'params': {'source': 'widget',
'id': 'http://us.i1.yimg.com/us.yimg.com/i/ww/beta/y3.gif',
'groupId': '@self',
'nolog': 'true',
'userId': '@viewer'},
…
bad_keypoints
- 1,382
- 2
- 23
- 45
1
vote
1 answer
Scraping by sending concurrent requests with python
I have python 3.4 and I installed requests and a few other necessary programs to web scrape. My problem is that I'd like to scrape about 7000 pages (just html/text), and don't want to do it all at once, I'd like to have some kind of delay so I don't…
thatandrey
- 287
- 5
- 16
0
votes
0 answers
How to process response as soon as it is available using grequests?
requests = (grequests.get(u) for u in urls)
responses = []
count = 1
for response in grequests.map(requests, stream=True):
response_list.append(response)
if rs == None:
print('{}. No Response'.format(count))
else:
…
Chiranjiv Sethi
- 3
- 1
0
votes
0 answers
Grequests on AWS Lambda "Gevent is required" Error
I want to use grequests on AWS Lambda. I created venv and reqs.txt and then pip install in a folder named python, then zipped the python folder and uploaded it to a Lambda Layer. After that, I am facing this error:
Gevent is required for…
Alper
- 152
- 1
- 7
0
votes
0 answers
using websockets like python grequests
Sorry if the problem seems obvious for most of you but I am quite new in using websockets and I can't get my way around this issue.
I want to receive multiple data at the same time on python via a specific API. So far I have been using the grequests…
rinma
- 1
- 1
0
votes
1 answer
How to send thousands of HTTP Requests using grerequests?
I need to request all the review pages for a company on Glassdoor, and in certain cases, there can be thousands of pages. I am trying to use grequests to do this, but I found that when I sent more than about 100 requests at once I would start to…
Dunc
- 59
- 4