How to get httpx.gather() with return_exceptions=True to complete the Queue of tasks when the exception count exceeds the worker count?

Question

I'm using asyncio in concert with the httpx.AsyncClient for the first time and trying to figure out how to complete my list of tasks when some number of them may fail. I'm using a pattern I found in a few places where I populate an asyncio Queue with coroutine functions, and have a set of workers process that queue from inside asyncio.gather. Normally, if the function doing the work raises an exception, you'll see the whole script just fail during that processing, and report the exception along with a RuntimeWarning: coroutine foo was never awaited, indicating that you never finished your list.

I found the return_exceptions option for asyncio.gather, and that has helped, but not completely. my script will still die after I've gotten the exception the same number of times as the total number of workers that I've thrown into my call to gather. The following is a simple script that demonstrates the problem.

from httpx import AsyncClient, Timeout
from asyncio import run, gather, Queue as asyncio_Queue
from random import choice


async def process_url(client, url):
    """
    opens the URL and pulls a header attribute
    randomly raises an exception to demonstrate my problem
    """
    if choice([True, False]):
        await client.get(url)
        print(f'retrieved url {url}')
    else:
        raise AssertionError(f'generated error for url {url}')


async def main(worker_count, urls):
    """
    orchestrates the workers that call process_url
    """
    httpx_timeout = Timeout(10.0, read=20.0)
    async with AsyncClient(timeout=httpx_timeout, follow_redirects=True) as client:
        tasks = asyncio_Queue(maxsize=0)
        for url in urls:
            await tasks.put(process_url(client, url))

        async def worker():
            while not tasks.empty():
                await tasks.get_nowait()

        results = await gather(*[worker() for _ in range(worker_count)], return_exceptions=True)
        return results

if __name__ == '__main__':
    urls = ['https://stackoverflow.com/questions',
            'https://stackoverflow.com/jobs',
            'https://stackoverflow.com/tags',
            'https://stackoverflow.com/users',
            'https://www.google.com/',
            'https://www.bing.com/',
            'https://www.yahoo.com/',
            'https://www.foxnews.com/',
            'https://www.cnn.com/',
            'https://www.npr.org/',
            'https://www.opera.com/',
            'https://www.mozilla.org/en-US/firefox/',
            'https://www.google.com/chrome/',
            'https://www.epicbrowser.com/'
            ]
    print(f'processing {len(urls)} urls')
    run_results = run(main(4, urls))
    print('\n'.join([str(rr) for rr in run_results]))

one run of this script outputs:

processing 14 urls
retrieved url https://stackoverflow.com/tags
retrieved url https://stackoverflow.com/jobs
retrieved url https://stackoverflow.com/users
retrieved url https://www.bing.com/
generated error for url https://stackoverflow.com/questions
generated error for url https://www.foxnews.com/
generated error for url https://www.google.com/
generated error for url https://www.yahoo.com/
sys:1: RuntimeWarning: coroutine 'process_url' was never awaited

Process finished with exit code 0

Here you see that we got through 8 of the total 14 urls, but by the time we reached 4 errors, the script wrapped up and ignored the rest of the urls.

What I want to do is have the script complete the full set of urls, but inform me of the errors at the end. Is there a way to do this here? It may be that I'll have to wrap everything in process_url() inside a try/except block and use something like aiofile to dump them out in the end?

Update To be clear, this demo script is a simplification of what I'm really doing. My real script is hitting a small number of server api endpoints a few hundred thousand times. The purpose of using the set of workers is to avoid overwhelming the server I'm hitting [it's a test server, not production, so it's not intended to handle huge volumes of requests, though the number is greater than 4 8-)]. I'm open to learning about alternatives.

You are creating 4 workers, and worker doesn't handle exceptions in its passed-in coroutine. After a worker raises, it's dead; so when you get 4 exceptions, your whole script is dead. You have gone to a lot of trouble to make this happen: creating a queue, writing/reading the queue, creating a limited number of workers to process the URLs. All of which defeats the inherent capability of gather() to handle an unlimited number of tasks and run them in parallel to a conclusion. Why did you do it this way? Why not just create a task for each URL and let gather sort it all out for you? — Paul Cornelius, Dec 30 '21 at 22:32
"Why did you do it this way?" Well, because I'm still learning how to use asyncio, and I'm capable of getting something wrong. ;-) I've updated my question with the additional info that my real script needs to throttle the number of tasks that gather() is working with in order to not overwhelm the single server that is the real target of my script. — Breaks Software, Dec 31 '21 at 14:22
@PaulCornelius your point is well taken that the worker doesn't handle exceptions. I've had a lot of difficulty finding any guidance on the proper way to do this in an asyncio context. Maybe that should be my real question? — Breaks Software, Dec 31 '21 at 14:27
Thanks for the clarification. I think your program is a fine way to throttle the number of simultaneous requests, so the exception handling becomes the real problem. I posted an answer. — Paul Cornelius, Dec 31 '21 at 23:15

score 0 · Accepted Answer · answered Dec 31 '21 at 23:09

The program design you have outlined should work OK, but you must prevent the tasks (instances of your worker function) from crashing. The below listing shows one way to do that.

Your Queue is named "tasks" but the items you place in it aren't tasks - they are coroutines. As it stands, your program has five tasks: one of them is the main function, which is made into a task by asyncio.run(). The other four tasks are instances of worker, which are made into tasks by asyncio.gather.

When worker awaits on a coroutine and that coroutine crashes, the exception is propagated into worker at the await statement. Because the exception isn't handled, worker will crash in turn. To prevent that, do something like this:

async def worker():
    while not tasks.empty():
        try:
            await tasks.get_nowait()
        except Exception:
            pass
            # You might want to do something more intelligent here
            # (logging, perhaps), rather than simply suppressing the exception

This should allow your example program to run to completion.

Thank you for the distinction between tasks and coroutines @Paul Cornelius. Getting the terminology straight in my head is important. It's also helpful for me to properly set my expectations for how all of this works, e.g. gather() isn't magic about handling exceptions. 8-) I have been focusing on handling exceptions inside process_url(), but you're right that it's also important to handle them inside of worker() so that those tasks won't die on me. — Breaks Software, Jan 01 '22 at 01:22
Marking this as accepted answer because it basically boiled down to needing to handle the exceptions and add code to pass along results, both positive and negative, from the coroutines through the gather() function. — Breaks Software, Jan 02 '22 at 14:15

How to get httpx.gather() with return_exceptions=True to complete the Queue of tasks when the exception count exceeds the worker count?

1 Answers1