0

Consider the following code which used concurrent.futures.ProcessPoolExecutor() to run a set of test code.

import time 
start =time.time()
def test_function(x):
    y=0;
    for ix in range (0,x):
        y=y+ix*2
    time.sleep(1);
    return y;

import concurrent.futures 

if __name__ == '__main__': 
    with concurrent.futures.ProcessPoolExecutor() as executor:
        results=[executor.submit(test_function,  ix) for ix in range(1,100000)]

        for ix in concurrent.futures.as_completed(results):
            print(ix.result())

On the Windows task manager, only two threads were actively running concurrently, with the rest of the threads not used. Though around the number of logical process's results were printed at once. This was kind of strange, since it implied that, even though the ProcessPoolExecutor() recognized the number of logical processors and is able to run the code in parallel, it did not exceed the single core utilization.

There were some related post mentioned similar issues, suggesting this might be connected to the Global Interpreter Lock.

  1. Python concurrent.futures using only 1 cpu core

However, there were other posts stating that ProcessPoolExecutor() is not constrained by GIL

  1. Is concurrent.futures a medicine of the GIL?

  2. How does ThreadPoolExecutor utilise 32 CPU cores for CPU bound tasks

How to make concurrent.futures.processpoolexecutor utilize all the physical cores in the computer, not just running several code in parallel in one core?

0 Answers0