Error during parallel computing of a function in Python 3.5, best approach?

Question

I have the next function that is called from a separated code and I want to run it in parallel using multiprocessing:

def GridParallel(vec_main, vec_egf, vec_lp):

    VR_list = []
    Synthetics = []

    for i in vec_main:

        ''' 
        List with VR vectors for each iteration of egf values, the size of the list is 
        given by the len of vec_egf
        '''

        temp_VR = []
        temp_Synth = []

        for j in vec_egf:

            for k in vec_lp:

                synth = Synthetic(Frequency=frequency_vector, LongPeriod_amp=k, 
                                        Corner_egf=j , Corner_main=i, 
                                         gamma=gamma, fall_off=fall_off)


                VR = Variance_Reduction(data=data, model=synth)

                temp_VR.append(VR)
                temp_Synth.append(synth)


        VR_list.append(temp_VR)
        Synthetics.append(temp_Synth)


    VR_array = np.asarray(VR_list)
    VR_max_values = np.amax(VR_array, axis=1) 


    # Corner index
    corner_index = VR_max_values.argmax()
    #corner_index = np.nonzero(VR_max_values == np.max(VR_max_values))

    # Best corner frequency for the main event
    Corner_main_shock = float(vec_main[corner_index])

    # Variace reduction corresponding to the main event
    Best_fit = float(VR_max_values[corner_index])


    # Indexing for extracting spectral ratios in gridsearch
    VR_max_values_index = VR_array.argmax(axis=1)

    Synthetics = np.asarray(Synthetics)


    N_Synthetics = []
    rows, cols, lenn = Synthetics.shape

    temp = np.arange(0, rows, 1)

    for ii in temp:
        spec = Synthetics[ii, VR_max_values_index[ii]]
        N_Synthetics.append(spec)


    N_Synthetics = np.asarray(N_Synthetics)

    return N_Synthetics, VR_max_values, Corner_main_shock, Best_fit

However, when I run:

pool = mp.Pool(processes=Ncores)

results = pool.apply_async(GridParallel, args=(vec_main, vec_egf, vec_lp, ))

pool.close()
results_ = results.get()
pool.join()

I get the next error:

AttributeError: Can't pickle local object 'GridSearchCorner.<locals>.GridParallel'

The input variables: vec_* are all numpy ndarrays (1d).

Also, how I can recover the return objects from the function?

I am trying to understand multiprocessing, so I apologize in advance if the question is too simple. Any help will be appreciated. For this I am working on python 3.5.

Thanks!!

Relevant [pickling-subclass-of-numpy-array](https://stackoverflow.com/questions/26598109/preserve-custom-attributes-when-pickling-subclass-of-numpy-array) — stovfl, Aug 08 '17 at 12:56

score 0 · Accepted Answer · answered Aug 09 '17 at 23:56

I have resolved this puzzle using Numba. In my script above, I am calling the functions Synthetic and Variance_Reduction, both of these functions are pure numpy / math functions. So using the @jit decorator makes the grid search really fast. For instance, before @jit it was taken around 80 s. to complete the task, with @jit the execution time was reduced to ~ 10 s.

Error during parallel computing of a function in Python 3.5, best approach?

1 Answers1