0

I am using joblib to parallel a for loop for my own function.

from joblib import Parallel, delayed
from my_function import my_case_study

result = Parallel(n_jobs=4)(delayed(my_case_study)(i) for i in range(100))

So my_case_study is the only function in my_function.py file, and it takes i as the hyperparameter. my_case_study calls a bunch of different model fitting algorithm contains in the other python files, which are imported at the top of my_function. my_function.py basicly looks like

from anotherfile import fun1
from anotherfile import fun2
def my_case_study(i):
     mse1 = fun1(i)
     mse2 = fun2(i)
     return (mse1,mse2)

But then I get the error message: A task gas failed to un-serialize. Please ensure that the arguments of the function are all picklable.

How to solve this? Thanks!!

Vickyyy
  • 197
  • 1
  • 9

1 Answers1

0

I found the solution in the following link helpful in my case:

https://github.com/joblib/joblib/issues/810

Don't know if there is any other better solution. Since the comments at the end mentioned that there might be some issues with that (didn't fully understand).

Vickyyy
  • 197
  • 1
  • 9