Need some advice on speeding up HTTP requests in python

Question

I need to send over 1 million HTTP requests and so far every option I've tried is just way too slow. I thought I could speed it up with aiohttp but that doesn't seem any faster than requests.

I was trying to do it with python but I'm open to other options as well.

Here is the code using both requests and aiohttp, any tips for speeding up the process?

requests code:

import requests

url = 'https://mysite.mysite:443/login'

users = [line.strip() for line in open("ids.txt", "r")]

try:
    for user in users:
        r = requests.post(url,data ={'username':user})
        if 'login.error.invalid.username' not in r.text:
            print(user, " is valid")
        else:
            print(user, " not found")
except Exception as e:
    print(e)

aiohttp code:

import aiohttp
import asyncio

url = 'https://mysite.mysite:443/login'
users = [line.strip() for line in open("ids.txt", "r")]   

async def main():
    async with aiohttp.ClientSession() as session:
        try:
            for user in users:
                payload = {"timeZoneOffSet": "240", "useragent": '', "username": user}
                async with session.post(url, data=payload) as resp:           
                    if 'login.error.invalid.username' not in await resp.text():
                        print(user, " is valid")
                    else:
                        print(user, " not found")
        except Exception as e:
            print(e)
            
loop = asyncio.get_event_loop()
loop.run_until_complete(main())

Are you trying to ddos/bruteforce something? – Jul 06 '20 at 21:40 — , Jul 06 '20 at 21:40
No, just a user enumeration script. – digital_alchemy Jul 07 '20 at 01:05 — digital_alchemy, Jul 07 '20 at 01:05

score 0 · Answer 1 · answered Jul 07 '20 at 07:15

You could use an asyncio.gather to collect results from a bunch of requests working in parallel.

Warning: code is just an example and is not tested.

import asyncio
from aiohttp import ClientSession

async def fetch(url, session, payload):
    async with session.post(url, data=payload) as resp:           
        if 'login.error.invalid.username' not in await resp.text():
            print(user, " is valid")
        else:
            print(user, " not found")

async def run(r):
    url = "http://your_url:8000/{}"
    tasks = []

    async with ClientSession() as session:
        for i in range(r):
            task = asyncio.ensure_future(fetch(url.format(i), session))
            tasks.append(task)

        responses = await asyncio.gather(*tasks)
        # you now have all response bodies in this variable

def print_responses(result):
    print(result)

loop = asyncio.get_event_loop()
future = asyncio.ensure_future(run(4))
loop.run_until_complete(future)

Need some advice on speeding up HTTP requests in python

1 Answers1