I've recently come across the problem where one needs to fetch a list of URLs as quickly as possible.
So naturally, I set up a small test to see what works best.
Approach 1 - asyncio
async def test_async():
async with httpx.AsyncClient() as client:
await asyncio.gather(*(fetch_async(client, symbol) for symbol in symbols))
async def fetch_async(client, symbol):
await client.get(
f"https://query1.finance.yahoo.com/v8/finance/chart/{symbol}.NS", timeout=None,
)
Approach 2 - ThreadPoolExecutor
async def test_threads():
with ThreadPoolExecutor(max_workers=len(symbols)) as pool, httpx.Client() as client:
loop = asyncio.get_event_loop()
await asyncio.gather(
*(
loop.run_in_executor(pool, fetch_sync_fn(client, symbol))
for symbol in symbols
)
)
def fetch_sync_fn(client, symbol):
def fn():
client.get(
f"https://query1.finance.yahoo.com/v8/finance/chart/{symbol}.NS",
timeout=None,
)
return fn
Results on a 2013 MacBook pro
In [3]: %timeit asyncio.run(test_threads())
1.41 s ± 87.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
In [4]: %timeit asyncio.run(test_async())
1.24 s ± 62.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Results on a digital ocean 5$ server
In [4]: %timeit asyncio.run(test_threads())
5.94 s ± 66.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
In [3]: %timeit asyncio.run(test_async())
10.7 s ± 97.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Results on a Google colab
%timeit loop.run_until_complete(test_threads())
1 loop, best of 3: 723 ms per loop
%timeit loop.run_until_complete(test_async())
1 loop, best of 3: 597 ms per loop
Questions
- What is the reason for this inconsistency? Why is there a different winner on the server vs local machine?
- Why are both tests slower on a server? Shouldn't a pure network task be faster on a server that has a faster network connection?
