Asyncio and aiohttp returning task instead of results

Question

I have a script to run parallel requests against an API within a class. However, the results I'm getting is basically a task instead of the actual results. Any reason why?

I mimicked the modified Client code on https://pawelmhm.github.io/asyncio/python/aiohttp/2016/04/22/asyncio-aiohttp.html.

import asyncio
from aiohttp import ClientSession

class Requestor:
    async def _async_request(self, url, session, sema_sz=10):
        sema = asyncio.Semaphore(sema_sz)

        async with sema:
            async with session.get(url) as response:
                req = await response.json()

        return req

    async def _async_chunk_request(self, url, chunks, headers=None, sema_sz=10):
        async with ClientSession(headers=headers) as session:
            futures = [asyncio.ensure_future(self._async_request(url.format(chunk), session, sema_sz)) for chunk in chunks]
            responses = asyncio.gather(*futures)
            await responses

    def get_request(self, url, chunks):
        loop = asyncio.get_event_loop()
        bulk_req = asyncio.ensure_future(self._async_chunk_request(url, chunks))
        loop.run_until_complete(bulk_req)

        return bulk_req

bulk_req is actually a Task variable and not the results and shows this in PyCharm, Task finished coro=<Requestor._async_chunk_request() done, defined at ...

When I debug, I see that req has a full and proper response value, so there's no issue with that. I feel like it's something to do with the actual gathering of the futures?

Looks like you are missing a closing paren for the `asyncio.ensure_future` call in the list comprehension. `...hunk), session, sema_sz)` <-- missing paren?? — wwii, Mar 01 '19 at 22:02
Seems like `futures` in `_chunk_request` is a list of tasks - `ensure_futures` wraps coroutines in tasks - does that make a difference? Without an api to test against someone will have to divine the issue from just reading your code. — wwii, Mar 01 '19 at 22:06
You are also calling `self._async_request` but that method is not shown in your example. Please take the time to read [mcve] and the other links found on that page. — wwii, Mar 01 '19 at 22:22
Sorry, those were typos on my end, I was on a different computer so I couldn't outright copy the code over. — srazzak, Mar 02 '19 at 14:04
In addition to things pointed out in the comments and the answer, please note that the code calls `run_until_complete` on the future, and then returns the future. It should be returning the return value of `run_until_complete`, which is the future's result. — user4815162342, Mar 02 '19 at 15:06

wwii · Accepted Answer · 2019-03-02T23:19:24.187

Your _chunk_request does not return anything.

async def _chunk_request(...):
    ...
    ...
    await responses

I made a toy example trying to mimic your process. If I ended _chunk_request like you did, i got the same result - a finished Task with No results. Changing _chunk_request to return something fixed it:

async def _chunk_request(...):
    ...
    ...
    return await responses

If you only need the return values from the tasks, get_request should return the result of the loop.run_until_complete() call.

My toy example

import asyncio
import random
from pprint import pprint

async def response(n):
    asyncio.sleep(random.choice([1,3,5]))
    return f'i am {n}'

async def _request(n):
    req = await response(n)
    #print(req)
    return req

async def _chunk_request(chunks):
    futures = [asyncio.ensure_future(_request(chunk)) for chunk in chunks]
    #pprint(futures)
    responses = asyncio.gather(*futures, return_exceptions=True)
    #pprint(responses)
    return await responses

def get_request(chunks):
    loop = asyncio.get_event_loop()
    bulk_req = asyncio.ensure_future(_chunk_request(chunks))
    return loop.run_until_complete(bulk_req)

In [7]: result = get_request(range(1,6))

In [8]: print(result)
['i am 1', 'i am 2', 'i am 3', 'i am 4', 'i am 5']

Worked! Looks like the main thing I was missing was returning the responses and pulling the results. Thank you! — srazzak, Mar 02 '19 at 14:04

Asyncio and aiohttp returning task instead of results

1 Answers1