Asyncio aiohttp populate list of HTML from list of URL

Question

I'm trying to get a list of HTML source code from a list of URLs asynchronously with asyncio and aiohttp but I get 2 Exceptions:

1. TypeError('An asyncio.Future, a coroutine or an awaitable is ' TypeError: An asyncio.Future, a coroutine or an awaitable is required Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x000001C92141F310>
2. raise RuntimeError('Event loop is closed') RuntimeError: Event loop is closed

This is my code:

import asyncio
import aiohttpasync 

def main():
    tasks = []
    html_list = []
    async with aiohttp.ClientSession() as session:
        for url in ['http://www.apple.com', 'http://www.google.cl']:
            async with session.get(url) as resp:
                tasks.append(html_list.append(await resp.read()))
        print(html_list[0])
        print(html_list[1])
        await(asyncio.wait(tasks))

if __name__ == '__main__':
    asyncio.run(main())

This code gets to print the html codes of ['http://www.apple.com', 'http://www.google.cl'] but immediatly after I get the aforementioned Exceptions.

Answer 1

There are a few things that aren't quite right with your example:

1. You're not running anything concurrently. When you await resp.read() in your for loop, this blocks main until the result comes back. Perhaps this is what you want, but you need to use asyncio.create_task to run your requests concurrently.

2. As pointed out, you don't need the task array at all because of point number one. You can just append to html_list.

3. You don't need to call asynio.wait because you're not awaiting any tasks or coroutines at this point.

You can resolve your direct issues by what you have done in the comments, but a version that runs concurrently looks like this:

import asyncio
import aiohttp

async def read_url(session, url):
    async with session.get(url) as resp:
        return await resp.read()


async def main():
    tasks = []
    async with aiohttp.ClientSession() as session:
        for url in ['http://www.apple.com', 'http://www.google.cl']:
            tasks.append(asyncio.create_task(read_url(session, url)))
        html_list = await asyncio.gather(*tasks)
        print(html_list[0])
        print(html_list[1])

if __name__ == '__main__':
    asyncio.run(main())

Here we define a coroutine read_url which gets the contents of a single url. Then in the loop, you create a task for reading the url and append it to the tasks list. Then you use asyncio.gather - this will wait for all tasks to finish concurrently.

As written, I'm unable to reproduce your RuntimeError: Event loop is closed error.