0

I have a computing task which effectively is running the same piece of code against a large number of datasets. I'm wanting to utilise large multicore systems(for instance 72 cores).

I'm testing it on a 2 core system

output_array = []
pool = ProcessPool(nodes=2)

i = 0
num_steps = len(glob.glob1(config_dir, "*.ini"))

worker_array = []

for root, dirs, files in os.walk(config_dir):

    for file_name in files:

        config_path = "%s/%s" % (config_dir, file_name)

        row_result = pool.apipe(individual_config, config_path, dates, test_section, test_type, prediction_service, result_service)
        worker_array.append(row_result)

with progressbar.ProgressBar(max_value=num_steps) as bar:

    bar.update(0)

    while len(worker_array) > 0:

        for worker in worker_array:

            if worker.ready():

                result = worker.get()
                output_array.append(result)
                worker_array.remove(worker)

                i = i + 1
                bar.update(i)

"individual_config" is my worker function.

My rationale for this code is to load the data to create all the tasks(2820 tasks), put the queue objects in a list, then poll this list to pick up completed tasks and put the results into an array. A progress bar monitors how the task is progressing.

Each individual_config task, running on it's own takes 0.3-0.5 seconds, but there's 2820 of them so that'd be about 20 odd minutes on my system.

When running in this new pathos multiprocessing processingpool configuration, it's getting one or two completed every 10-15 seconds. The task is projected to take 20 hours. I'd expect some overhead of the multiprocessing, so not getting a double speed with two cores processing, but this seems to be something wrong. Suggestions?

David Findlay
  • 1,296
  • 1
  • 14
  • 30
  • Do you get the same performance with raw multiprocessing as you do above with pathos? In your real world case are you using class methods? If so, I believe the whole class is sometimes serialized and sent to the worker process, so large class methods or attributes can make this kind of work slow. Also, isn't the `.get()` method blocking? You may want to check out one of the async methods like `imap` – duhaime Aug 18 '18 at 01:26
  • I wasn't able to get it working with raw multiprocessing as my task function takes multiple arguments. While there seems to be some workarounds to make the standard multiprocessing library run with multiple arguments in functions, it seems painful. I moved to pathos initially in the hope that it'd make this easier. – David Findlay Aug 18 '18 at 01:28
  • My worker function is a standard method, not a class method. It does receive several hundred kb of data in the arguments though. – David Findlay Aug 18 '18 at 01:29
  • if that's the case I'd go back to multiprocessing, as I've found pathos more difficult to work with and less performant generally. Can you just pass an array of args to your worker function? That will work just fine in multiprocessing. Do the workers need to write to that several hundred KB of data, or do they need read only access? If the latter, let them read from the master parent's scope, it will save time. If I were you I'd use imap with multiprocessing, passing in an array of args. – duhaime Aug 18 '18 at 01:30
  • I'm calling worker.ready() to see if the result is available before calling get, so I was hoping that'd mean it only blocks when the data is actually available to get. – David Findlay Aug 18 '18 at 01:30
  • Thanks for that, I've moved back to doing it with multiprocessing and an array of args. It seems to work and is about 3 times slower than without any multiprocessing stuff, that seems reasonable although i'd have hoped for better – David Findlay Aug 18 '18 at 01:43
  • amen, glad to help. There is some overhead like you said. 72 cores should still give you some solid performance though! – duhaime Aug 18 '18 at 01:44

0 Answers0