0

I'm trying to have a grequest spawn additional grequests and I'm finding that the second-level grequest callbacks are not executed. The usecase is to view the first page of results from an api, use it to calculate how many additional pages there are to visit, and then create new grequests for those pages. Here is the relevant code snippet:

def handle_page(response, **kwargs):
    # this code is not executed
    print(response.url)
    print(response.status_code)

def handle_tag(response, **kwargs):
    tag = url_to_tag_regex.findall(response.url)[0]
    print(tag)
    jsonObj = response.json()
    totalCount = jsonObj.get("total_count")
    totalPages = int(ceil(totalCount / 48.0))
    unsent_tag_reqs = (grequests.get(api_url_format % (tag, page),
                                     callback=handle_page,
                                     session=requests_session) for page in xrange(totalPages))
    grequests.map(unsent_tag_reqs, size=__CONCURRENT_LIMIT__)

if __name__=="__main__":
    unsent_requests = (grequests.get(api_url_format % (tag, 0),
                                     callback=handle_tag,
                                     session=requests_session) for tag in tagList)
    grequests.map(unsent_requests, size=__CONCURRENT_LIMIT__)
sajattack
  • 803
  • 1
  • 9
  • 23
  • Have you confirmed that the requests are even made or that handle_tag is even called? – Ian Stapleton Cordasco Jun 28 '16 at 13:35
  • handle_tag runs. handle_page does not. It gets to the nested generator but does not send the nested requests iirc. – sajattack Jun 28 '16 at 15:07
  • So it's not calling handle_page, but have you used wireshark to see if the requests are even sent over the network? – Ian Stapleton Cordasco Jun 28 '16 at 18:27
  • Considering it's got ~20 000 requests to send and the program stops after the outer nest finishes I believe the requests in the inner nest are not sent. – sajattack Jun 28 '16 at 23:03
  • So, I think you're not understanding how asynchronous requests work. If you make a new request that doesn't block in `handle_tag`, and you don't have anyway of waiting for the program to exit before that request completes, the program won't wait. It doesn't say "OH! There are requests in flight... better not exit!" it just says "There's nothing left to do... exiting" You need a way to wait for those (whatever way you choose). – Ian Stapleton Cordasco Jun 29 '16 at 12:28
  • Oh, right. That makes sense. But why would the handle_tag callback work as expected and handle_page wouldn't? – sajattack Jun 29 '16 at 21:22
  • Oh they're coming out as NoneType. – sajattack Jun 29 '16 at 21:34

0 Answers0