Multiprocessing with complex function

Question

I want to create a Multiprocess for a quite complex function in python: I have tested this function with a less complex code like this:

from joblib import Parallel, delayed, parallel_backend
from joblib import load, dump

def print_hello(hallo, tschüß, rechnen,i):
    print(i)
    print(hallo[2])
    print (tschüß)
    rechnen = rechnen +i
    hallo2 = pd.DataFrame(hallo)
    hallo2.to_csv('./hallo'+str(i)+'.csv')
    hallo1 = pd.read_csv('./hallo'+str(i)+'.csv')
    return rechnen

hallo = ['hallo', 'hi', 'hey']
tschüß = 'tschüß'    
with parallel_backend('threading'):
test = Parallel()(delayed(print_hello)(hallo, tschüß, rechnen, i) for i in range(10))

print(test)

This is working quit nice. However i get the following Error-Code:

joblib.my_exceptions.TransportableException: TransportableException

...

joblib.my_exceptions.JoblibTypeError: JoblibTypeError

...

TypeError: sum_row() missing 1 required positional argument: 'i'

When I want to make my complex function works, which looks like this:

def sum_row(count_series, path, folder, files_1, files_2, files_3, path_raw, i):
    print(i)
    df1 = pd.read_csv(path_raw + files_1[i], sep=',', low_memory=False)
    df2 = pd.read_csv(path_raw + files_2[i], sep=',', low_memory=False)
    df3 = pd.read_csv(path_raw + files_3[i], sep=',', low_memory=False)

    ##do some operations with those files and create df_test

    df_test.to_csv(path + folder + files_export[i])

    return 0

with parallel_backend('threading'):
    test = Parallel()(delayed(sum_row)(count_series, path, files_1, files_2, files_3,  path_raw, i) for i in range(len(files_1)))

you're missing the folder argument it seems when you call your function. — mgracer, Feb 15 '18 at 09:32
@mgracer Do you somehow know why it is quite slow and just uses a small percentage of the CPU power? It is using many CPUs but all just a little bit? — Mimi Müller, Feb 15 '18 at 10:12
use debugging tools to figure it out there are python modules for debugging performance — mgracer, Feb 15 '18 at 10:54
fyi --> use multiprocessing instead of threading as backend then it is working. And if there are more CPUs available than Jobs then it runs into a error so modify the n_job in Parallel() based on the count of your jobs. — Mimi Müller, Feb 16 '18 at 08:18

mgracer · Accepted Answer · 2018-02-15T09:47:04.297

1

The reason you are getting the error is that you're missing the folder argument when calling the function.

test = Parallel()(delayed(sum_row)(count_series, path, folder, files_1, files_2, 
files_3,  path_raw, i) for i in range(len(files_1)))

edited Feb 15 '18 at 09:47

answered Feb 15 '18 at 09:40

mgracer

175
1
11

Multiprocessing with complex function

1 Answers1