PicklingError: Can't pickle : attribute lookup builtin.function failed (multiprocessing)

Question

I am going to implement a kind of time-series clustering method, which my optimization solution is based on frank-wolfe algorithm. I would like to parallelize it by multiprocessing library in python, but I face the following problem:

Traceback (most recent call last):
File "/home/ridha/Downloads/pycharm-2018.1.2/helpers/pyde/pydev_run_in_console.py", line 52, in run_file

pydev_imports.execfile(file, globals, locals)  # execute the script

File "/home/ridha/Desktop/TICC-FW/TICC-master/example.py", line 8, in <module>
(cluster_assignment, cluster_MRFs) = ticc.fit(input_file=fname)

File "/home/ridha/Desktop/TICC-FW/TICC-master/TICC_solver.py", line 111, in fit
train_cluster_inverse)

File "/home/ridha/Desktop/TICC-FW/TICC-master/TICC_solver.py", line 295, in optimize_clusters
val = optRes[cluster].get()

File "/home/ridha/anaconda2/lib/python2.7/multiprocessing/pool.py", line 572, in get
raise self._value

PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

and my code is:

probSize = self.window_size * size_blocks

lamb = np.zeros((probSize, probSize)) + self.lambda_parameter

S = np.cov(np.transpose(D_train))
empirical_covariances[cluster] = S

# rho = 1

lamb_vec = lamb.flatten()
lamb_vec_abs = np.absolute(lamb_vec)
s_vec = S.flatten()

aub = np.identity(probSize*probSize)
bub = np.ones(probSize*probSize)

def loss(x):

    theta = np.reshape(x, (probSize, probSize))
    det_theta = -np.linalg.det(theta)

    return det_theta + (np.matmul(s_vec, x)) + (
        np.matmul(lamb_vec_abs, np.absolute(x)))

solver = FrankWolfe(f=loss,
                    g=None, A_ub=aub, b_ub=bub, A_eq=None, b_eq=None, bounds=None,
                    tol=1.0E-12, gradient_delta=1.0E-10, iteration_limit=1000)

# solver.verbose = True

# apply to process pool

optRes[cluster] = pool.apply_async(solver)

I would appreciate any help

can you please correct to indentation of your example ? Is `loss` a function within a function ? — rocksportrocker, Sep 06 '18 at 11:31
You might want to adapt this solution to your problem: https://stackoverflow.com/questions/8804830/python-multiprocessing-pickling-error#answer-24673524 — rocksportrocker, Sep 06 '18 at 12:05
Your stacktrace and sample code do not match at all. Can you please provide the code that produces your error? — MisterMiyagi, Sep 06 '18 at 12:55

score 0 · Answer 1 · answered Sep 06 '18 at 12:28

0

I had the same issue with multiprocessing a while ago, I am now using a custom multiprocessing library named pathos but you would have to compile the library yourself and after that install it with pip. After this whole trouble I found out that lambda cannot be pickled. I only see some parameter names using it in you code but I can't see anything calling the function itself, if there is, try replacing it. Also avoid any method or function defined inside another method or function.

answered Sep 06 '18 at 12:28

Alexandru Martalogu

273
2
12

First of all, I want to know that can we parallelize simplex method?! because, frank-wolfe algorithm optimizes my problem by solving it via simplex method! – reza ghadiri Sep 07 '18 at 06:08

PicklingError: Can't pickle : attribute lookup __builtin__.function failed (multiprocessing)

1 Answers1

PicklingError: Can't pickle : attribute lookup builtin.function failed (multiprocessing)