Python grequests issue

Question

I am trying to make calls to several URLs at the same time using the grequests library for Python. The problem is that I do not quite understand the logic of grequests. Below is a sample (edited version) of my code:-

respArray = []
response = []
sessionvar = requests.Session()
sessionvar.trust_env = False
for each in range(0,len(urls)):
        response.append(grequests.get(urls[each],session=sessionvar,cookies=cookiesArray[each]))
        eachresp = grequests.map(response)
 for r in eachresp:
    respArray.append(r.json())

return respArray

My respArray returns each individual array which was returned from the URLs. When I run that array, it's as if each one is running on its own loop, and not concurrently. I am not getting how can I get it to run concurrently so that I get faster results.

Answer 1

This code:

for each in range(0,len(urls)):
        response.append(grequests.get(urls[each],session=sessionvar,cookies=cookiesArray[each]))
        eachresp = grequests.map(response)
 for r in eachresp:
    respArray.append(r.json())

Is effectively sending each request sequentially. You send the url and wait for grequests to send it in each loop iteration.

Basically it looks like:

• Loop iteration, wait for request
• Loop iteration, wait for request

etc.

You need to follow how their documentation suggests:

# Build a list of unsent requests
requests = (grequests.get(url) for url in urls)
# Send them all at once
results = grequests.map(requests)
return [r.json() for r in results]

This will build your list of unsent requests and then send them all together. In other words:

• Loop iteration, create unsent request
• Loop iteration, create unsent request
• ...
• Send all requests simultaneously