2

I am using the grequests to asynchronously download data from a website using the same url but different parameters.

For example,

unsent_requests = []
for param in params: # assume params is a list containing different parameters or query strings
    unsent_requests.append(grequests.get(url = url, params = param))

responses = grequests.map(unsent)

How can I possibly get to know which response from responses belongs to which request from unsent_requests? Or are the responses in the same order as the unsent requests?

PS: response.url does not give any clue because a completely different url returns.

idrisadetunmbi
  • 877
  • 2
  • 9
  • 22

2 Answers2

4

Responses are in the same order as the requests, as shown in the usage example:

>>> reqs = [
...    grequests.get('http://httpbin.org/delay/1', timeout=0.001),
...    grequests.get('http://fakedomain/'),
...    grequests.get('http://httpbin.org/status/500')]
>>> grequests.map(reqs, exception_handler=exception_handler)
Request failed
Request failed
[None, None, <Response [500]>]
engineerC
  • 2,808
  • 17
  • 31
  • 1
    Are you absolutely sure about this? – vks Nov 28 '16 at 06:57
  • 3
    This is incorrect, the requests are processed in order, but the results are guaranteed not in order. – dcruwys Jun 30 '17 at 17:02
  • 3
    Look at the source code link. The responses array is built from the requests array, in order. While it is true that the requests may be in flight in random order, and they may finish in random order, *the responses array is a one-for-one in-order match to the requests array*. – engineerC Jul 10 '17 at 16:19
3

No, they are not, the very nature of asynchronous requests means that despite being processed in order, the actual responses are not guaranteed to be in order.

This is why the response contains the originalrequest, which you can see by iterating through the result of map and printing response.request.url.

dcruwys
  • 107
  • 2