speed up test with asyncio

Question

Here is my code

@pytest.mark.asyncio
async def test_async():
    res = []
    for file_id in range(100):
        result = await get_files_async(file_id)
        res.append(result)
    assert res == [200 for _ in range(100)]


async def get_files_async(file_id):
    kwargs = {"verify": False, "cert": (cert, key)}
    resp = requests.request('GET', url, **kwargs)
    return resp.status_code

The timing from pytest shows it takes 118 sec to finish, which is very close to the test that sends request to url in sequence. Is there any improvement that can speed up this test? Thanks.

score 1 · Accepted Answer · answered Nov 04 '21 at 08:37

You cannot speed this up using async, since you are using requests, which is a sync pkg, so each call stops the even loop.

You can either switch to running the requests in threads or switch to an async pkg like httpx or aiohttp

If you do switch to a different pkg, change the test_async to this to run the requests in parallel

@pytest.mark.asyncio
async def test_async():
    tasks = []
    for file_id in range(100):
        tasks.append(asyncio.create_task(get_files_async(file_id)))
    res = await asyncio.gather(*tasks)
    assert res == [200 for _ in range(100)]

Thank you for your answer. I switch to aiohttp and it does speed up the test. — Jeffrey S., Nov 06 '21 at 00:19

speed up test with asyncio

1 Answers1