3

I have a loop which does a bunch of CPU intensive calculations and appends the result to a list, per iteration.

How can I have that working in parallel. On C# there are concurrent containers, how does it work in ipyparallel?

From the ipyparallel documentation:

Python’s builtin map() functions allows a function to be applied to a sequence element-by-element. This type of code is typically trivial to parallelize.

http://ipyparallel.readthedocs.io/en/latest/direct.html#parallel-map

So it's a matter of using the map function to have it running on parallel, but how can I append the results to a list? Are there concurrent containers here?

So, what I have now is something like this:

results = []
for element in list:
    outcome = very_heavy_computation_function(element)
    results.append(outcome)

How can I do this in parallel?

Abel Dantas
  • 380
  • 2
  • 17
  • Have a look at the python documentation: https://docs.python.org/3/library/concurrency.html. Your questions are probably too broad SO. If you have a specific issue, feel free to ask. – jpp Feb 23 '18 at 00:44
  • My question isn't too broad. Thank you for the link to the python docs. I want to append an element to a list that is being populated on a for loop. I think it may be one of the most trivial types of chunks to parallelize. Will update my question with more details though, thanks. – Abel Dantas Feb 23 '18 at 13:53

1 Answers1

3

You can achieve it in the following way.

Function foo represents calculations in the loop, parameters represents the data you want to iterate over if any. Even though foo sleeps for 10s the whole loop takes exactly 10s instead of 40s because I have 4 engines in my environment and functions run parallelly on the engines. A LoadBalancedView provides dynamic load balancing to evenly distribute work among engines.

from ipyparallel import Client

rc = Client()
view = rc.load_balanced_view()

def foo(param):
    import time
    time.sleep(10)
    return param*2

parameters = list(range(4))
async_results = []
for p in parameters:
    async_result = view.apply_async(foo, p)
    async_results.append(async_result)

rc.wait_interactive(async_results)

results = [ar.get() for ar in async_results]
print(results)

Output:

   4/4 tasks finished after   10 s
done
[0, 2, 4, 6]
Tomi
  • 89
  • 6
  • I've got terrible scoping problems when I attempt to employ this example. Whenever I try to reference variables I get 'is not defined' errors. Ugh. – Keith Jun 16 '22 at 05:37