Python: running n number of functions in parallel

Question

I would like to call N number of python functions in parallel, but am not sure how to go about this.

I have read this thread and have implemented some of the code already.

from multiprocessing import Process
def func1():
  print 'func1: starting'
  for i in xrange(10000000): pass
  print 'func1: finishing'

def func2():
  print 'func2: starting'
  for i in xrange(10000000): pass
  print 'func2: finishing'

def runInParallel(*fns):
  proc = []
  for fn in fns:
    p = Process(target=fn)
    p.start()
    proc.append(p)
  for p in proc:
    p.join()

if __name__ == '__main__':
    runInParallel(func1, func2)

That's the code from the link above. I've adapted that code to work with my situation without issue.

So with this code, I can obviously run two functions in parallel. I could just add more functions manually, (func1, func2, func3, func4, ...). That seems messy and unpythonic.

Is there a way to re-do this code to call any number of functions in parallel?

have a look at this other post, maybe can be useful: [Post](http://stackoverflow.com/questions/7207309/python-how-can-i-run-python-functions-in-parallel) — Carlo 1585, May 18 '16 at 16:18
@Carlo1585 Yes, that's the post I linked to. From that I figured out how to run multiple functions in parallel... But if I want to run say 20 functions in parallel, I don't want to have to write the code for all 20 functions. I'd like to be able to call one function 20 times, but in parallel. — Flux Capacitor, May 18 '16 at 16:22

score 2 · Accepted Answer · edited May 23 '17 at 10:28

2

You can pass arguments to the Process class (docs) for example:

p = Process(target=func, args=('func1', ))

and func looks like:

def func(name):
    print '%s: starting' % name
    for i in xrange(10000000): pass
    print '%s: finishing' % name

As a side note you might want to be using Thread instead of Process - see here for a comparison: https://stackoverflow.com/a/3046201/419115

edited May 23 '17 at 10:28

Community

1
1

answered May 18 '16 at 16:43

Michoel

834
5
16

score 1 · Answer 2 · answered May 18 '16 at 16:45

you can use multiprocessing.Pool.map or imap

from multiprocessing import Pool

def f(x):                    # the function to be executed in parallel
    for i in xrange(10000000): pass

pool = Pool(processes=4)     # 4 worker processes
print pool.map(f, range(20)) # 20 jobs, a list will be returned when all are finished

alternatively you can get results as soon as they're finished using

for result in pool.imap(f, range(20))
    print result

Python: running n number of functions in parallel

2 Answers2