4

Hi There i'm struggling with my I/O bound app to make it fast enough for potential users

im fetching an X number of urls say 10 for example, using MULTI THREADING with 1 thread for each URL

but that takes too long i've ran Cprofile on my code and i see that the bottleneck is in "{method 'acquire' of '_thread.lock' objects} "

in the Cprofile result i noticed that the method 'acquire' is called 9 Times per Thread

Can anybody please shed some light on how i can reduce the number of calls per Thread here is a sample code:

url_to_get = ["https://api.myip.com","https://api.myip.com","https://api.myip.com","https://api.myip.com",
              "https://api.myip.com","https://api.myip.com","https://api.myip.com","https://api.myip.com",
              "https://api.myip.com","https://api.myip.com"]

def fetch(url):
    with requests.get(url,proxies=proxy) as response:
        print(response.text)
        
def main():
    with ThreadPoolExecutor(max_workers=10) as executor:
        executor.map(fetch, url_to_get)
       
        

if __name__ == '__main__':
    import cProfile, pstats
    profiler = cProfile.Profile()
    profiler.enable()
    main()
    profiler.disable()
    stats = pstats.Stats(profiler).sort_stats('tottime')
    stats.print_stats(10)

Cprofile Results:

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       90    3.581    0.040    3.581    0.040 {method 'acquire' of '_thread.lock' objects}
       10    0.001    0.000    0.001    0.000 C:\Users\MINOUSAT\AppData\Local\Programs\Python\Python38-32\lib\threading.py:1177(_make_invoke_excepthook)
       10    0.001    0.000    0.001    0.000 {built-in method _thread.start_new_thread}
       10    0.000    0.000    0.028    0.003 C:\Users\MINOUSAT\AppData\Local\Programs\Python\Python38-32\lib\concurrent\futures\thread.py:193(_adjust_thread_count)
       20    0.000    0.000    0.025    0.001 C:\Users\MINOUSAT\AppData\Local\Programs\Python\Python38-32\lib\threading.py:270(wait)
       21    0.000    0.000    0.000    0.000 C:\Users\MINOUSAT\AppData\Local\Programs\Python\Python38-32\lib\threading.py:222(__init__)
       10    0.000    0.000    0.028    0.003 C:\Users\MINOUSAT\AppData\Local\Programs\Python\Python38-32\lib\concurrent\futures\thread.py:158(submit)
       32    0.000    0.000    0.000    0.000 {built-in method _thread.allocate_lock}
       10    0.000    0.000    0.001    0.000 C:\Users\MINOUSAT\AppData\Local\Programs\Python\Python38-32\lib\threading.py:761(__init__)
       10    0.000    0.000    0.025    0.002 C:\Users\MINOUSAT\AppData\Local\Programs\Python\Python38-32\lib\threading.py:540(wait)

Thank you so much

Driss ait
  • 66
  • 3
  • I'm trying to figure this out too. I can't seem to resolve the reference "_thread.lock"... (e.g. `import _thread; _thread.lock` -> `module '_thread' has no attribute 'lock'`. – cowlinator Jan 25 '23 at 01:36

0 Answers0