2

EXAMPLE:

The below code is capable of getting multiple web results with grequests but why on earth does it fail with "making simple api calls"

CODE:

import grequests

links = [
    'https://api.github.com/users?since=135',
    'http://www.google.com',
    'https://api.github.com/users?since=135',
    'http://www.google.com'
]

a = (grequests.get(i) for i in links)
p = grequests.map(a)
print(p)

Why the heck is the output not: (Response [200], Response [200], Response [200], Response [200])

But: (None, Response [200], None, Response [200])

...just skipping the api calls in purpose ):

Brad Solomon
  • 38,521
  • 31
  • 149
  • 235
Nomeh Uchenna Gabriel
  • 104
  • 10

2 Answers2

1

If the requests library (as used by grequests) returns None as the response (for whatever reason), grequests lets you handle that however you'd like, by calling an exception handler callback function that you create and then pass to map.

From grequests.py (in the map method):

    for request in requests:
        if request.response is not None:
            ret.append(request.response)
        elif exception_handler and hasattr(request, 'exception'):
            ret.append(exception_handler(request, request.exception))
        elif exception_handler and not hasattr(request, 'exception'):
            ret.append(exception_handler(request, None))
        else:
            ret.append(None)

What is happening here? Before this block executes, grequests will have raised all of the requests, and now we're looping through the results. For each request:

  • If you get a response, return that.
  • If an exception handler is registered and the request has an exception defined, call the handler and pass it the request and exception.
  • If an exception handler is registered and the request does not have an exception defined, call the handler and pass it the request.
  • If the response was None, but no handler is registered, return None

The last case results in data loss, but it's preventable by using a callback that properly handles the exception. What exactly exception_handler does is something you need to define, then include like this:

response = grequests.map(request_getters, exception_handler=my_handler)

what the handler does is up to you, but maybe this would be helpful:

MAX_PARALLEL_REQUESTS = 2

links = [
    'https://api.github.com/users?since=135',
    'http://www.google.com',
    'https://api.github.com/users?since=135',
    'http://www.google.com'
]

def my_handler(request, exception):
    links.append(request.url)
    print(f"exception thrown by grequests: \n{exception}")
    return request

while links:
    a = (grequests.get(links.pop(0)) for _ in range(MAX_PARALLEL_REQUESTS))
    p = grequests.map(a, exception_handler=my_handler)
    print(p)

This is pop a fixed number of URL's off the links list each iteration of the while loop. If any of those requests should fail, my_handler is called, which adds the failed URL back to the links list for reprocessing.

Z4-tier
  • 7,287
  • 3
  • 26
  • 42
-2

finaly i've realised that all i needed was to hardcode it with loops instead of using grequests.

Here

:

import requests, time

links = [
    'https://api.github.com/users?since=135',
    'http://www.google.com',
    'https://api.github.com/users?since=135',
    'http://www.google.com'
]

sd = []

for i in links:
    try:
        d = requests.get(i)
        time.sleep(1)
        sd.append(d)
    except:
        sd.append(False)

print(sd)

#note that time.sleep(*) is only optional.

...don't know why grequests behaves so.

Nomeh Uchenna Gabriel
  • 104
  • 10