0

When using concurrent.futures in Python for large data sets ( 9x200,000x4 np.floats ) I've noticed that CPU usage is low (13% ~ equivalent to 1 core being used) for the beginning. However after a while it shoots up to what I expect for multiprocessing (80-90%). Here's a snippet of my code, if interested.

sections = np.arange(0,9,1)
section_result = []
sectioned_results = []


if __name__ == "__main__":
    plt.close()
    with concurrent.futures.ProcessPoolExecutor() as executor:
    
        generatorlist = [executor.map(makeimage,sections) for _ in range(num_particles)]
  for generator in generatorlist:
      for item in generator:
          section_result.append(item)

Does anyone know the purpose of this? It seems to be an exponential rise in time taken on this 1 core doing something as my number of particles increases. My first thought was allocation of memory as I anticipate this run to take up around 1-1.5GB, but there doesn't seem to be anything in python docs about this process, and I wonder whether I've implemented the module incorrectly. I've tested this on relatively low data sets (10,000 - 100,000) and have definitely seen a rise in duration of 1 core being used.

Many thanks

A

ARLI
  • 1

1 Answers1

0

It all depends on what your program is actually doing. For example, if all you run in your subprocesses is a while True: pass then I'd expect you would see your usage rise to 100% immediately.

For example, any IO bound operation (reading from disk, waiting for an API response) would cause you to see low CPU usage while your program is waiting on the result of that operation.

wakey
  • 2,283
  • 4
  • 31
  • 58
  • Sadly there is no IO bound operation nor any API use in my code. I'll try and be as clear as possible with the structure of the code. The general purpose of the code is to simulate 10^6 particles being randomly scattered within a box. This uses numpys random generator to generate a random starting position and velocity, and then uses s = ut until it crosses the box boundary and then reflects. – ARLI Nov 17 '20 at 16:14