Fetch HEAD request's status asynchronously in aiohttp

Question

Question is regarding aiohttp libriary usage.

My goal here is to check list of urls by sending bunch of HEAD requests, potentially asynchronously ,and eventually create dict of url: status pairs.

I am new in asyncio and stuff and I found a lot of examples where people use GET requests to fetch html ,for example ,and they use await resp.read() or await resp.text() and it works fine but with HEAD request I don’t have body, I just have header, that's it. If I try to await resp.status or resp itself as an object – it does not work as they are not awaitable.

Code below works only synchronously step by step and I can’t figure out how to make it run asynchronously. Seems like whatever i do with status turns code to sync mode somehow...

I would be glad to see your ideas.

Thanks.

import asyncio
import aiohttp

urls_list = [url1,  url2,  url3, etc, etc, etc, ]
status_dict = {}


async def main():

    async with aiohttp.ClientSession() as session:
        for individual_url in urls_list:
            async with session.head(individual_url) as resp:
                status_dict.update({url: resp.status})

    
asyncio.run(main())

Even if you could await the status, it wouldn't help you because "await" doesn't automatically make the code parallel, it means just the opposite ("wait until the result is available"). To make the code parallel, use `create_task` or `gather` as shown in Alex Noname's answer. — user4815162342, Jul 20 '20 at 08:57

score 4 · Accepted Answer · answered Jul 20 '20 at 08:30

You can you asyncio.gather:

import asyncio
import aiohttp

urls_list = ["https://google.com", "https://yahoo.com", "http://hello123456789.com"]
status_dict = {}


async def head_status(session, url) -> dict:
    async with session.head(url) as resp:
        return {url: resp.status}


async def main():
    async with aiohttp.ClientSession() as session:
        statuses = await asyncio.gather(*[head_status(session, url) for url in urls_list], return_exceptions=True)
    for a in statuses:
        if not isinstance(a, Exception):
            status_dict.update(a)

asyncio.run(main())

Fetch HEAD request's status asynchronously in aiohttp

1 Answers1