How to post group of requests to 2 urls with aiohttp

Question

I have 2 URLs and 60k+ requests. Basically, I need to post every request to both URLs, then compare their responses, but not to wait for the response to post another request.

I've tried to do it with aiohttp and asyncio

import asyncio
import time
import aiohttp
import os

from aiofile import AIOFile

testURL = ""
prodURL = ""

directoryWithRequests = ''
directoryToWrite = ''

headers = {'content-type': 'application/soap+xml'}

i = 1


async def fetch(session, url, reqeust):
    global i
    async with session.post(url=url, data=reqeust.encode('utf-8'), headers=headers) as response:
        if response.status != 200:
            async with AIOFile(directoryToWrite + str(i) + '.xml', 'w') as afp:
                await afp.write(reqeust)
                i += 1
        return await response.text()


async def fetch_all(session, urls, request):
    results = await asyncio.gather(*[asyncio.create_task(fetch(session, url, request)) for url in urls])
    return results


async def asynchronousRequests(requestBody):
    urls = [testURL, prodURL]
    global i
    with open(requestBody) as my_file:
        body = my_file.read()

    async with aiohttp.ClientSession() as session:
        htmls = await fetch_all(session, urls, body)
        # some conditions

async def asynchronous():
    try:
        start = time.time()
        futures = [asynchronousRequests(directoryWithRequests + i) for i in os.listdir(directoryWithRequests)]
        for future in asyncio.as_completed(futures):
            result = await future

        print("Process took: {:.2f} seconds".format(time.time() - start))

    except Exception as e:
        print(str(e))

if __name__ == '__main__':
    try:
        # AsyncronTest
        ioloop = asyncio.ProactorEventLoop()
        ioloop.run_until_complete(asynchronous())
        ioloop.close()
        if i == 1:
            print('Regress is OK')
        else:
            print('Number of requests to check = {}'.format(i))

    except Exception as e:
        print(e)

I believe that the code above works, but it creates N futures, where the N equals to the number of request files. This brings to sort of ddos because the server can't response to that number of requests at the same time.

score 0 · Answer 1 · answered Jul 04 '19 at 12:36

Found suitable solution. Basically it's just 2 async tasks:

tasks = [
                postRequest(testURL, client, body),
                postRequest(prodURL, client, body)
            ]
await asyncio.wait(tasks)

It's not the same performance as the code in the question with afortable number of requests, but as least it doesn't ddos the server that much.

How to post group of requests to 2 urls with aiohttp

1 Answers1