GRequests allows you to use Requests with Gevent to make asyncronous HTTP Requests easily.
GRequests allows you to use requests with gevent to make asynchronous HTTP Requests easily.
Questions tagged [grequests]
139 questions
2
votes
1 answer
Asynchronous requests don't work using GRequests
I'm working on a program. It reads urls from a file and then makes requests. But handlers don't get called.
Here is my code:
import grequests
def main():
async_list =[]
for site in extractsites():
action_item =…
zython
- 1,176
- 4
- 22
- 50
2
votes
1 answer
Asynchronous scraping with Python: grequests and Beautifulsoup4
I am trying to scrape this site . I managed to do it by using urllib and beautifulsoup. But urllib is too slow. I want to have asynchronous requests because the urls are thousands. I found that a nice package is grequests.
example:
import…
dimosbele
- 381
- 3
- 19
2
votes
2 answers
Mysterious error when using django server with grequests
Currently, I am running a vagrant server on Ubuntu 14.04 and I test all my django modules by using the simple python manage.py runserver 0.0.0.0:8000
Since I am connecting to the django webserver using chrome through http://localhost:8000 and the…
AlanSTACK
- 5,525
- 3
- 40
- 99
2
votes
1 answer
Applying retry on grequests in Python
I have a list of URLs for which I want to do a HTTP Get request using Python's grequests module.
Some of the URLs don't return an OK status, and in that case, I want to retry that URL.
I can do this using a Queue which stores all such URLs that have…
xennygrimmato
- 2,646
- 7
- 25
- 47
2
votes
2 answers
Are grequests responses in the same order as the requests?
I am using the grequests to asynchronously download data from a website using the same url but different parameters.
For example,
unsent_requests = []
for param in params: # assume params is a list containing different parameters or query strings
…
idrisadetunmbi
- 877
- 2
- 9
- 22
2
votes
1 answer
Python: grequest and request give different response
My original task: Using Trello API, get data through HTTP GET requests. Run requests and process responses asynchronously, if possible. The API provider uses "https://" URL I access to with some key and token.
Tools I used:
Python 2.7.10 | Anaconda…
PMvn
- 53
- 7
2
votes
1 answer
Is it possible to sleep without yielding?
I want to send two asynchronous requests using grequests.send with a short but exact delay (say 20 ms) between them. I only want to handle the responses after both requests have been sent.
Putting a time.sleep between the two sends doesn't work…
akxlr
- 1,142
- 9
- 23
2
votes
1 answer
Use of grequests with a single request
I recently came across some internal code at my workplace that uses the grequests library to make its HTTP requests.
However, as I was looking through the code, I noticed that the program only ever sends a single request at a time. For instance,…
therealrootuser
- 10,215
- 7
- 31
- 46
2
votes
1 answer
How to send multiple http requests python
I am using python request.post() to send data to a remote database. How should I send more than one request(around 20-30) on same URL using different data using python ?
Also, will sequential work fine for this case or do I need to make the requests…
bitgeeky
- 319
- 1
- 4
- 15
2
votes
0 answers
Asynchronous File Upload with Flask API using Python Twisted?
I currently have a python script that runs on my local machine that uploads some files by calling a basic API I built using Flask that's on a remote machine. The script basically iterates thru a list of file paths, checks for some criteria and…
django-d
- 2,210
- 3
- 23
- 41
1
vote
1 answer
How to avoid 429 error when using grequests?
I want to make a parser using "grequests", but as expected, due to many requests, i get an error 429.
import grequests
sites = [f"https://fantlab.ru/work{i}" for i in range(1, 10)]
response = (grequests.get(url, timeout=5) for url in sites)
resp…
Mop
- 53
- 5
1
vote
1 answer
How to Scrape Product Pages using Python grequests and BeautifulSoup
from bs4 import BeautifulSoup
import grequests
import pandas as pd
# STEP 1: Create List of URLs from main archive page
def get_urls():
urls = []
for x in range(1,3):
…
MarkWP
- 177
- 6
1
vote
1 answer
How to parse the response from Grequests faster?
I want to webscraping multiple urls and parse quick as possible but the for loop is not too faster for me, have a way to do this maybe with asynchronous or multiprocessing or multithreading?
import grequests
from bs4 import BeautifulSoup
links1 =…
JONH
- 17
- 4
1
vote
0 answers
How to use grequest to automatically reconnect after grequest timeout
How does grequests do something like'requests' time out auto reconnect.
import time
import requests
from requests.adapters import HTTPAdapter
for i in range(1,1000):
s = requests.Session()
s.mount('http://', HTTPAdapter(max_retries=3))
…
Relaxing
- 33
- 4
1
vote
1 answer
Why does my for loop only check the last item?
I've built a async'ed program thats going to check if a element exists on multiple paths of a website.
The program has a base url, that will get different paths of the domain to check, which are located in a json file (name.json).
If the element I'm…
user14108046