GRequests allows you to use Requests with Gevent to make asyncronous HTTP Requests easily.
GRequests allows you to use requests with gevent to make asynchronous HTTP Requests easily.
Questions tagged [grequests]
139 questions
3
votes
1 answer
Multiple http get and post
The task is:
1)send an http get to url based on a parameter
2)Modify the response based on the same parameter
3)send an http post to url based on the same parameter
I am currently doing this through requests library, but it takes a lot of time to…
vks
- 67,027
- 10
- 91
- 124
3
votes
1 answer
Getting AttributeError while importing grequests in Python 3.4
I've been trying to use grequests for asyncronous HTTP requests in my Python project, however I got an AttribteError and have been searching for solution for hours.
Here's my testing code:
import grequests
urls = ['htt://google.com',…
Mori Nakamura
- 33
- 2
3
votes
1 answer
Too many urls in greqests.map causes gevent.hub.LoopExit only on mac
In my code (https://github.com/chembl/chembl_webresource_client/blob/master/chembl_webresource_client/web_resource.py#L98)
I'm calling grequests.map:
ret = grequests.map(rs, size=50)
with rs as list of URLs. If the size of that list is larger than…
mnowotka
- 16,430
- 18
- 88
- 134
3
votes
0 answers
Python - grequests socket.gaierror: [Errno -2] Name or service not known)
I am using grequests for doing a several requests to crunchbase API.
Unfortunately, they dont provide a way to query data based on the updated time of the company, so in order to get the most recently funded companies I have to scrape all their…
Manuel G
- 1,523
- 1
- 21
- 34
3
votes
1 answer
Python Grequests xml responses
Quick question with grequests since the documentation for it is rather sparse.
What is the best way to return xml reponse from the request sent? I'm having trouble finding a way to get a response back other than the status codes. Could someone…
user2758113
- 1,001
- 1
- 13
- 25
3
votes
0 answers
Python Grequests + multiprocessing
I'm trying to make async http calls using grequests from within a Pool from the multiprocessing library. I'm encountering a problem that suggests Grequests and multiprocessing are possibly incompatible with each other; specifically, calling…
kfan
- 1,641
- 1
- 13
- 16
2
votes
1 answer
Passing async function as argument
I am trying to use use grequests to make a single HTTP call asynchronously.
All goes find until I try to call a function (handle_cars) to handle the response.
The problem is that the function is an async function and it I don't know how to await it…
Tiago
- 347
- 1
- 3
- 12
2
votes
1 answer
Extracting Text From Asynchronus Request Using grequests
I am trying to extract the text part from the request that I made through grequest library but I am unable to figure out how can I do so.
If we use Requests Library I would do
r = requests.get('www.google.com')
htmls.append(r.text)
Now if I am…
Lopez
- 461
- 5
- 19
2
votes
1 answer
Python Webscraping: grequests vs. mult-threaded requests?
I'm trying to make web-scraper I'm writing in Python faster.
Currently I fire up a set amount of scraper threads, create a queue with a list of URLs I want to scrape and let them dequeue entries so they can scrape.
grequests states it's asynchronous…
Omrii
- 397
- 3
- 14
2
votes
1 answer
Even the gevent and monkey patching added top of the imports its giving error
Even the gevent and monkey patching added top of the imports its giving error: MonkeyPatchWarning: Patching more than once will result in the union of all True parameters being patched curious_george.patch_all(thread=False, select=False) please…
Sudhakar Donkena
- 41
- 1
- 7
2
votes
1 answer
Why does grequests.get() return None?
I'm trying to fetch automatically the MAC addresses for some vendors in Python. I found a website that is really helpful, but I'm not being able to access its information from Python. When I run this:
import grequests
rs = (grequests.get(u) for u…
Tendero
- 1,136
- 2
- 19
- 34
2
votes
2 answers
Why does 'API' requests return 'None' in grequests while 'normal web scraping' gives a 200 response
EXAMPLE:
The below code is capable of getting multiple web results with grequests but why on earth does it fail with "making simple api calls"
CODE:
import grequests
links = [
'https://api.github.com/users?since=135',
…
Nomeh Uchenna Gabriel
- 104
- 10
2
votes
3 answers
How to slow down asynchrounous API calls to match API limits?
I have a list of ~300K URLs for an API i need to get data from.
The API limit is 100 calls per second.
I have made a class for the asynchronous but this is working to fast and I am hitting an error on the API.
How do I slow down the asynchronous,…
RustyShackleford
- 3,462
- 9
- 40
- 81
2
votes
2 answers
grequests - too many open files error
I'm trying to perform about 100k GET requests and parse the response body of each request. I thought grequests would be a good way to go, but I'm getting errors related to 'too many open files'. Here's the code:
import grequests
with…
user8967637
2
votes
0 answers
handle url-redirects in asynchronous code
Recently I decided to port a project from using requests to using grequests to make asynchronous HTTP Requests instead for efficiency. One problem I face now
is that in my previous code, in cases where there was a redirect I could handle it using…
game0ver
- 1,250
- 9
- 22