Hi There i'm struggling with my I/O bound app to make it fast enough for potential users
im fetching an X number of urls say 10 for example, using MULTI THREADING with 1 thread for each URL
but that takes too long i've ran Cprofile on my code and i see that the bottleneck is in "{method 'acquire' of '_thread.lock' objects} "
in the Cprofile result i noticed that the method 'acquire' is called 9 Times per Thread
Can anybody please shed some light on how i can reduce the number of calls per Thread here is a sample code:
url_to_get = ["https://api.myip.com","https://api.myip.com","https://api.myip.com","https://api.myip.com",
"https://api.myip.com","https://api.myip.com","https://api.myip.com","https://api.myip.com",
"https://api.myip.com","https://api.myip.com"]
def fetch(url):
with requests.get(url,proxies=proxy) as response:
print(response.text)
def main():
with ThreadPoolExecutor(max_workers=10) as executor:
executor.map(fetch, url_to_get)
if __name__ == '__main__':
import cProfile, pstats
profiler = cProfile.Profile()
profiler.enable()
main()
profiler.disable()
stats = pstats.Stats(profiler).sort_stats('tottime')
stats.print_stats(10)
Cprofile Results:
ncalls tottime percall cumtime percall filename:lineno(function)
90 3.581 0.040 3.581 0.040 {method 'acquire' of '_thread.lock' objects}
10 0.001 0.000 0.001 0.000 C:\Users\MINOUSAT\AppData\Local\Programs\Python\Python38-32\lib\threading.py:1177(_make_invoke_excepthook)
10 0.001 0.000 0.001 0.000 {built-in method _thread.start_new_thread}
10 0.000 0.000 0.028 0.003 C:\Users\MINOUSAT\AppData\Local\Programs\Python\Python38-32\lib\concurrent\futures\thread.py:193(_adjust_thread_count)
20 0.000 0.000 0.025 0.001 C:\Users\MINOUSAT\AppData\Local\Programs\Python\Python38-32\lib\threading.py:270(wait)
21 0.000 0.000 0.000 0.000 C:\Users\MINOUSAT\AppData\Local\Programs\Python\Python38-32\lib\threading.py:222(__init__)
10 0.000 0.000 0.028 0.003 C:\Users\MINOUSAT\AppData\Local\Programs\Python\Python38-32\lib\concurrent\futures\thread.py:158(submit)
32 0.000 0.000 0.000 0.000 {built-in method _thread.allocate_lock}
10 0.000 0.000 0.001 0.000 C:\Users\MINOUSAT\AppData\Local\Programs\Python\Python38-32\lib\threading.py:761(__init__)
10 0.000 0.000 0.025 0.002 C:\Users\MINOUSAT\AppData\Local\Programs\Python\Python38-32\lib\threading.py:540(wait)
Thank you so much