Using grequests to send a pool of requests, how can I get the response time of each individual request?

Question

I am using the grequests python library to send GET requests asynchronously to our server.

I cant figure out how to get the server response time for each individual request within the pool of requests being sent out?

unsentrequests=(grequests.get(u) for u in self.urls) # make a pool of requests
responses=grequests.map(unsentrequests) # send the requests asynchronously

To get the start time of a request-response pair I could do the following:

grequests.get(u,headers={'start':time.time())
print responses[0].request.headers['start_time']

But how can I grap the time that the response was received at?

You have three questions [webapps.se] that have some good answers. Perhaps you can take some time and evaluate them, as you do on [so]. — Jacob Jan Tuinstra, Jun 25 '13 at 20:50
Try this one : https://stackoverflow.com/questions/34656037/python-grequets-to-get-time-for-each-http-response-individually — darkdefender27, Nov 07 '17 at 06:57

score 14 · Accepted Answer · answered Jan 10 '13 at 13:47

14

grequests, like requests, support an hook keyword argument where you can assign a function that does something with the response object:

def do_something(response):
    print response.status_code

unsentrequests=(grequests.get(u, hooks = {'response' : do_something}) for u in self.urls) 
responses=grequests.map(unsentrequests)

I prefer to use requests directly in a loop using gevent to have a more explicit code:

import gevent.monkey
gevent.monkey.patch_socket()
from gevent.pool import Pool
import requests

def check_urls(urls):

    def fetch(url):
        response = requests.request('GET', url, timeout=5.0)
        print "Status: [%s] URL: %s" % (response.status_code, url)

    pool = Pool(20)
    for url in urls:
        pool.spawn(fetch, url)
    pool.join()

answered Jan 10 '13 at 13:47

raben

3,060
5
32
34

Why do you put the fetch() function as a sub-function to check_urls()? (I'm just learning python, so I'm trying to figure out what the adv would be, versus just putting it in the more general scope) – Travis Leleu Mar 28 '14 at 18:44
1

There is no reason, it is just that I'm a bad programmer ;-) Jokes aside you could rewrite my code defining `fetch` outside the main function. – raben Apr 15 '14 at 22:14
Doesn't this print the status only after each 20th request? Is there a way to print the status after each request with a pool size > 1? – boadescriptor Apr 28 '15 at 22:28

Using grequests to send a pool of requests, how can I get the response time of each individual request?

1 Answers1

Linked