0

The idea of the program is to check for domains/subdomains living (via http/https protocol) in the subdomains.txt file.

I did this by using HEAD requests to domains/subdomains and receiving the response status code. If the status code is available, the domain or subdomain is live. (load_url_http function)

To speed up the program, I used concurrent.futures.ThreadPoolExecutor with a number of threads of 200 However, even after increasing the number of threads to 300, the program's speed isn't much improved.

I want an improvement in my program to be able to send thousands of requests at once. Below is part of my source code:

python-request-multil.py

import time

import requests
import concurrent.futures


def load_url_http(protocol: str, domain: str, timeout: int = 10):
        try:
            conn = requests.head(protocol + "://" + domain, timeout=timeout)
            return conn.status_code
        except Exception:
            return None


#--- main ---#
start_time = time.time()

worker = 400
protocol = "http"
timeout = 10

print("Number of worker:", worker)

with concurrent.futures.ThreadPoolExecutor(max_workers=worker) as executor:
    # The file object that the subdomain lives on will be written to
    file_live_subdomain = open("live_subdomains.txt", "a")
    
    # load domain/subdomain list from file
    URLS = open("subdomains.txt", "r").read().split("\n")
    URLS_length = len(URLS)
    
    # Count the number of live subdomains
    live_count = 0
    
    # Start the load operations and mark each future with its URL
    future_to_url = {
        executor.submit(load_url_http, protocol, url, timeout): url for url in URLS
    }
    
    for i, future in zip(range(URLS_length), concurrent.futures.as_completed(future_to_url)):
        url = future_to_url[future]
        print(f"\r-->  Checking live subdomain.........{i+1}/{URLS_length}", end="")
        try:
            data = future.result()
            
            # If `load_url_http` returns any status code
            if data != None:
                # print(f'{protocol}://{url}:{data}')
                live_count = live_count + 1
                file_live_subdomain.write(f"\n{protocol}://" + url)
        except Exception as exc:
            print(exc)
    print(f"\n[+] Live domain: {live_count}/{URLS_length}", end="")
    file_live_subdomain.close()

print("\n--- %s seconds ---" % (time.time() - start_time))

Run:

┌──(quangtb㉿QuangTB)-[/mnt/e/DATA/Downloads]
└─$ python3 python-request-multil.py
Number of worker: 100
-->  Checking live subdomain.........1117/1117
[+] Live domain: 344/1117
--- 67.41670227050781 seconds ---

┌──(quangtb㉿QuangTB)-[/mnt/e/DATA/Downloads]
└─$ python3 python-request-multil.py
Number of worker: 200
-->  Checking live subdomain.........1117/1117
[+] Live domain: 344/1117
--- 54.6825795173645 seconds ---

┌──(quangtb㉿QuangTB)-[/mnt/e/DATA/Downloads]
└─$ python3 python-request-multil.py
Number of worker: 300
-->  Checking live subdomain.........1117/1117
[+] Live domain: 339/1117
--- 54.186068058013916 seconds ---

┌──(quangtb㉿QuangTB)-[/mnt/e/DATA/Downloads]
└─$ python3 python-request-multil.py
Number of worker: 400
-->  Checking live subdomain.........1117/1117
[+] Live domain: 344/1117
--- 54.19181728363037 seconds ---
Quang Tran
  • 3
  • 2

1 Answers1

2

In python multithreading doesn't actually run in parallel, all the threads run under 1 process and that process runs only on 1 core (of the cpu).

You can create as many threads as you want and is won't solve the problem and can actually make it worse, because those 300 threads are running on 1 cpu core but that 1 cpu core can only run 1 command at a time, so what ends up happening is that the cpu core needs to run a few commands on 1 thread and then switch to another thread and run a few commands on that other thread and then switch to another one and etc... the switching action between threads takes resources and in that time your program code doesn't run. So in the end if you open to many threads your cpu core will spend more time on switching between threads than on executing your program.

What you can do to actually run your code simultaneously is to open a few processes instead of threads using the multiprocessing library, and then your code will run on a number of cores, and the same thing here, don't open hundreds of processes, open only a few, I recommend on openings 1 process for each core your cpu has, the multiprocessing library has a built-in function the returns the number of cores your cpu has:

import multiprocessing
print(multiprocessing.cpu_count())

note that because of the fact that the processes actually run your code simultaneously your prints can interfere with each other so you will need to use multiprocessing.Lock() and do something like this:

import multiprocessing

lock = multiprocessing.Lock()

lock.acquire()
print("something")
lock.release()

do lock.acquire() before each print and lock.release() after (if you won't release the lock your program will be stuck) and this will make sure that your prints won't get in each other.

EDIT:

in your case because of the timeout it will actually be better to open a few processes and in each process something like 20 threads.

because if the process is waiting 10 seconds for each failed address it will end up being slower than opening a lot of threads

so for your case the fastest way I can think of is opening a few processes and in each process open 20 to 30 threads

You can try something like this:

import multiprocessing
import random
import threading
import time
import requests


MAX_NUMBER_OF_PROCESSES = multiprocessing.cpu_count()
MAX_NUMBER_OF_THREADS_IN_EACH_PROCESS = 30
PROTOCOL = "http"
TIMEOUT = 4
START_TIME = time.time()


def load_url_http(protocol: str, domains: list[str], timeout: int = 10):
    with open("live_subdomains.txt", "a") as live_domains_file:
        for domain in domains:
            try:
                conn = requests.head(protocol + "://" + domain, timeout=timeout)
                if conn.status_code is not None:
                    live_domains_file.write(f"{protocol}://{domain}\n")
            except Exception:
                pass
    return


def create_threads_for_process(protocol: str, domains: list[str], timeout: int = 10):
    # create threads
    threads_list = []
    threads_urls = {}
    start = 0
    number_of_threads_to_open = len(domains) if len(domains) < MAX_NUMBER_OF_THREADS_IN_EACH_PROCESS \
        else MAX_NUMBER_OF_THREADS_IN_EACH_PROCESS
    for i in range(1, number_of_threads_to_open + 1):
        # distribute the work of this process evenly between all the threads
        if i != number_of_threads_to_open:
            threads_urls[i] = domains[start:(len(domains) // number_of_threads_to_open) * i]
            start = (len(domains) // number_of_threads_to_open) * i
        else:
            threads_urls[i] = domains[start:]
        # create and start the thread
        thread = threading.Thread(target=load_url_http,
                                  args=(protocol, threads_urls[i], timeout,),
                                  daemon=True)
        thread.start()
        threads_list.append(thread)
    # wait for all threads to finish
    while threads_list:
        for thread in threads_list:
            if not thread.is_alive():
                threads_list.remove(thread)
        time.sleep(0.8)


def main():
    with open("live_subdomains.txt", "w") as file:
        file.write("")
    with open("subdomains.txt", "r") as file:
        urls = file.read().split("\n")
    random.shuffle(urls)  # shuffle the urls list
    # create the processes
    processes_list = []
    processes_urls = {}
    start = 0
    number_of_processes_to_open = len(urls) if len(urls) < MAX_NUMBER_OF_PROCESSES else MAX_NUMBER_OF_PROCESSES
    for i in range(1, number_of_processes_to_open + 1):
        if i != number_of_processes_to_open:
            # give each process an even amount of work
            processes_urls[i] = urls[start:(len(urls) // number_of_processes_to_open) * i]
            start = (len(urls) // number_of_processes_to_open) * i
        else:
            # the last process will get a bit more / a bit less in
            # case len(urls) isn't dividable by number_of_processes_to_open
            processes_urls[i] = urls[start:]
        # create the process, start it and add to processes list
        process = multiprocessing.Process(target=create_threads_for_process,
                                          args=(PROTOCOL, processes_urls[i], TIMEOUT,),
                                          daemon=True)
        process.start()
        processes_list.append(process)
    # wait for all processes to finish
    while multiprocessing.active_children():
        time.sleep(0.8)
    # print result
    with open("live_subdomains.txt", "r") as live_urls_file:
        live_count = len(live_urls_file.read().split("\n")) - 1  # -1 empty line at the end
    print(f"\n[+] Live domain: {live_count}/{len(urls)}", end="")
    print("\n--- %s seconds ---" % (time.time() - START_TIME))


if __name__ == '__main__':
    main()

in this code I opened 1 process for each core the cpu has, and in each process I opened 30 threads, also you don't need to wait 10 seconds for a replay something like 5 seconds is enough especially when you use HEAD and not GET.

I run your code and compared to mine, your code took 35 seconds to finish, mine took 25 seconds and that is on 1117 urls, the bigger the url list the more significant it will be.

Omer Dagry
  • 537
  • 4
  • 9