GRequests allows you to use Requests with Gevent to make asyncronous HTTP Requests easily.
GRequests allows you to use requests with gevent to make asynchronous HTTP Requests easily.
Questions tagged [grequests]
139 questions
0
votes
1 answer
Python SQL not inserting data for the whole list of URLs
I am using the grequest module to use multiple URLs.
However, my issue here is that I'm only getting data inserted into my database from 1 of the URLs and not each one in the List.
Question:
How can I solve this?
import urllib.request as…
fazzerking
- 23
- 7
0
votes
1 answer
Extracting links from website with selenium bs4 and python
Okay so.
The heading might seem like this question has already been asked but I had no luck finding an answer for it.
I need help with making link extracting program with python.
Actually It works. It finds all elements on a webpage. Takes…
Mileta Dulovic
- 1,036
- 1
- 14
- 33
0
votes
2 answers
Making Array Of Requests Using Requests Library In Python
I am using the requests library to map an array of requests after I get an array from another API request. I am using a loop for the requests but I sure there is a better way to do this because this API request can have 500+ items, so finishing this…
Demetrius
- 449
- 1
- 9
- 20
0
votes
1 answer
Multithreading requests without a URL list
I'm currently trying to make my requests faster by multithreading them but I'm not sure how to do it the way I want. I know about grequests but they seem to require a URL list. I have code with a starting number contained in URL and would like all…
kreny
- 18
- 4
0
votes
2 answers
How to follow redirect in grequests?
How to follow redirect in grequests ?
In requests library we can simply use : r = requests.get(url, allow_redirects=True) to allow redirect .
is there anything in grequests ?
i have checked https://pypi.org/project/grequests/ but did not found…
Pranav Kumar
- 33
- 1
- 5
0
votes
1 answer
Additional arguments in Flask grequests hook
I am having issue in passing additional parameter to grequests using a hook, Its working in a standalone app (non flask) but its not with flask (flask integrated server) Here is my code snippet.
self.async_list = []
for url in self.urls:
…
user8781522
- 71
- 3
- 11
0
votes
1 answer
How to stagger asynchronous API calls to prevent Max retries with grequests library?
I have a list of ~250K urls for an API that I need to retrieve.
I have made a class using grequests that works exactly how I want it to except, I think it is working too fast because after running through the entire list of URLs I get…
RustyShackleford
- 3,462
- 9
- 40
- 81
0
votes
1 answer
How to send batches of URLs into grequest?
I have a list of ~300K API URLs that I want to call and get data from:
lst = ['url.com','url2.com']
If I subset my list down to say 5 urls grequest handles the request perfectly. However when I pass in the full ~300K URLs I get error:
Problem:…
RustyShackleford
- 3,462
- 9
- 40
- 81
0
votes
0 answers
Is it possible to speed up a web scraper with multiple computers?
Is there any way to speed up a web-scraper by having multiple computers contribute to processing a list of urls? Like computer A takes urls 1 - 500 and computer B takes urls 501 - 1000, etc. I am looking for a way to build the fastest possible web…
Random Programmer
- 29
- 2
0
votes
1 answer
grequests to get .csv file from API
I need to get multiple .csv files from SharePoint.
If I make this request via Postman
https://mycompany.sharepoint.com/teams/a/g/_api/web/GetFolderByServerRelativeUrl('Data%20Sources\')/Files('sharepoint_test.csv')/$value
With…
user3871
- 12,432
- 33
- 128
- 268
0
votes
1 answer
grequests.post() is returning None
>>> payload = {'a': 'b'}
>>> url = 'http://someurl/'
>>> req_ = [grequests.post(url, session=requests.session, data=payload)]
>>> r = grequests.map(req_)
>>> r
[None]
am I doing something wrong?
doniyor
- 36,596
- 57
- 175
- 260
0
votes
1 answer
Using libraries that use patched and non-patched version of another library
I guess this question (Import not Working) means the following and it is kind of interesting:
Library A and B are actually grequest and multiprocessing. (I heard) grequest patches socket but multiprocessing can't use the patched version.
I want to…
hamster on wheels
- 2,771
- 17
- 50
0
votes
1 answer
Python 2.7 qrequests.map on separate thread gets response but grequests.send does not
I have a python program that creates a separate process for a HttpServer to run on. Then I make a request to that HttpServer using grequests.map on a separate thread or grequests.send with a threading.Pool.
When I use grequests.map on a separate…
Matthew Hoggan
- 7,402
- 16
- 75
- 140
0
votes
1 answer
grequests returning 'None' type even after using exception_handler
I am using grequests module for making async requests. While testing the following piece of code is showing ANOMALY based on TIMEOUT value:
>>>…
kappa101
- 71
- 1
- 7
0
votes
1 answer
Wikipedia Queries + Grequests
I'm trying to query ~50 Wikipedia pages. I've been using the requests package to make GET requests, but I've been working on implementing grequests as I hear it has much better performance.
The performance improvement is really quite minimal for…
John Kim
- 15
- 2