1

I want to translate a huge matlab model to python. Therefor I need to work on the key functions first. One key function handles parallel processing. Basically, a matrix with parameters is the input, in which every row represents the parameters for one run. These parameters are used within a computation-heavy function. This computation-heavy function should run in parallel, I don't need the results of a previous run for any other run. So all processes can run independent from eachother.

Why is starmap_async slower on my pc? Also: When i add more code (to test consecutive computation) my python crashes (i use spyder). Can you give me advice?

import time
import numpy as np
import multiprocessing as mp
from functools import partial

# Create simulated data matrix
data = np.random.random((100,3000))
data = np.column_stack((np.arange(1,len(data)+1,1),data))

def EAF_DGL(*z, package_num):
    sum_row = 0
    for i in range(1,np.shape(z)[0]):
        sum_row = sum_row + z[i] 
    func_result = np.column_stack((package_num,z[0],sum_row))
    return func_result


t0 = time.time()
if __name__ == "__main__":
    package_num = 1
    help_EAF_DGL = partial(EAF_DGL, package_num=1)
    with mp.Pool() as pool:
        #result = pool.starmap(partial(EAF_DGL, package_num), [(data[i]) for i in range(0,np.shape(data)[0])])
        result = pool.starmap_async(help_EAF_DGL, [(data[i]) for i in range(0,np.shape(data)[0])]).get()
    pool.close() 
    pool.join()
t1 = time.time()
calculation_time_parallel_async = t1-t0
print(calculation_time_parallel_async)

t2 = time.time()
if __name__ == "__main__":
    package_num = 1
    help_EAF_DGL = partial(EAF_DGL, package_num=1)
    with mp.Pool() as pool:
        #result = pool.starmap(partial(EAF_DGL, package_num), [(data[i]) for i in range(0,np.shape(data)[0])])
        result = pool.starmap(help_EAF_DGL, [(data[i]) for i in range(0,np.shape(data)[0])])
    pool.close() 
    pool.join()
t3 = time.time()
calculation_time_parallel = t3-t2
print(calculation_time_parallel)
Lathano
  • 23
  • 2

0 Answers0