Stack Overflow
Stack Overflow

Questions tagged [grequests]

GRequests allows you to use Requests with Gevent to make asyncronous HTTP Requests easily.

GRequests allows you to use requests with gevent to make asynchronous HTTP Requests easily.

139 questions
1
vote
1 answer

Asynchronous Webscraping with beautiful soup, grequests

I am trying to webscrape some information from a list of urls, and would like to do it asynchronously to save time. I am able to get the information I would like when I do so synchronously, and the relevant part of that code is essentially…
TNoms
  • 81
  • 4
1
vote
0 answers

Define a variable once in an asynchronous loop? Python grequests

I have this code that I am trying to make asynchronous def m1(): links={ #my list of links } rs = (grequests.get(u) for u in links) res = grequests.map(rs) for r in res: z = 0 soup =…
casualpy
  • 11
  • 3
1
vote
2 answers

Asynchronous URL un-shortening in Python

I am currently trying to implement a feature into my program that will detect and unshorten any URL shorteners, including bit.ly and old goo.gl links (now no longer existent). I have found a few articles, and I am going to discuss my current…
Generic Nerd
  • 308
  • 1
  • 7
  • 19
1
vote
0 answers

"pip install gevent" on windows gives IndexError

I'm trying to pip install grequests on Windows 10, but it errors out when it attempts to install gevent. I'll be grateful for any clues on how to resolve this. (FWIW, I also tried pip --no-cache-dir install gevent, amongst other variants. They all…
Matt Muggeridge
  • 465
  • 3
  • 11
1
vote
0 answers

Difference in using grequests with chunks of addresses or all at once?

I try to download the /home.htm file of around 6000 http addresses. For speed I tried to use grequests to send them all at once but I only get around 200 answers, most of them give a connection refused error. When I split the addresses in chunks of…
RubiBubi
  • 11
  • 2
1
vote
2 answers

How to map grequests response with request?

I wrote a python code to send async requests using grequests, how to check which request's response is this ? are responses are in same order as they are sent ? Here is my code : import grequests import time start_time = time.time() q_values =…
Pranav Kumar
  • 33
  • 1
  • 5
1
vote
1 answer

How to throttle GET requests from a list of URLs

I have a list of ~250000 urls, that I need to get data from an API. I have created a class using the grequests library to make asynchronous calls. However the API limit is 100 calls per second, which grequest surpasses. Code using…
RustyShackleford
  • 3,462
  • 9
  • 40
  • 81
1
vote
1 answer

How to print same dictionary object from multiple urls with grequest?

I have a list of URLs that all use the same json structure. I am trying to pull specific dictionary objects from all of the URLs at once with grequest. I am able to do it with one URL, though I am using request: import requests import json main_api…
Jim Carlson
  • 51
  • 9
1
vote
0 answers

Python grequests gets None responses

I'm crawling this website using grequests (at a pretty slow pace), and at the first few hundreds of requests I get mostly 200 HTTP responses, but at a certain point, all I get are None responses(without any raised exception). The weirdest part is…
omer
  • 69
  • 7
1
vote
1 answer

install grequests but error: "No module named grequests"

https://i.stack.imgur.com/QUGyz.png It says grequests is succesfull installed. But when i want to import it. I get a error saying "No module named grequests" How can i fix this? System = OS 10.11.6
Joch1009
  • 21
  • 1
  • 3
1
vote
1 answer

How to send 1k POST http requests per second with data using Python?

I need to do stress test for my API. I modified that example to send POST instead of HEAD, also I put some data to my request: count = int(sys.argv[1]) url = "http://api.example.com/" test_id = datetime.now().strftime("%H%M%S%f") data = [] for i in…
Sawl Stone
  • 11
  • 1
  • 3
1
vote
1 answer

How can I execute batches of requests simultaneously in python?

I'm using the Reddit api to upvote some subreddits. I'd like to upvote multiple subreddits simultaneously, using the praw library to interact with the reddit api. I want to upvote the 27 "hot" and "rising" posts of /r/bookclub: subreddit =…
ian-campbell
  • 1,605
  • 1
  • 20
  • 42
1
vote
0 answers

Is it possible to do a python async request with certificate from memory instead of a file?

I am trying to write a tool to generate a lot of requests from Python. The requests require a certificate to be attached to each request. Meaning I am doing something like: requests.post('/path/', cert=('cert.pem', 'private.key')) …
mrdvlpr
  • 526
  • 4
  • 20
1
vote
1 answer

grequests - first couple of responses 200 then 403

I have collected links to wanted people from Interpol website . There are about 10k links. Scraping one by one takes hours so I am looking for the way to do it asynchronously with grequests. This is the preview of my links…
pawelty
  • 1,000
  • 8
  • 27
1
vote
1 answer

Python grequests issue

I am trying to make calls to several URLs at the same time using the grequests library for Python. The problem is that I do not quite understand the logic of grequests. Below is a sample (edited version) of my code:- respArray = [] response =…
Sindhujit
  • 51
  • 9
1 2 3
…
9 10