3

I keep having an issue when executing a function multiple times at once using the multiprocessing.Pool class. I am using Python 3.8.3 on Windows 10 with PyCharm 2017.3.

The function I am executing is opening and serialising excel files from my harddisk to custom objects which I want to iterate through later on. The error always occurs after the last execution of the function. Here is what it says:

multiprocessing.pool.MaybeEncodingError: Error sending result: '[<IntegListe.IntegrityList object at 0x037481F0>, <IntegListe.IntegrityList object at 0x03D86CE8>, <IntegListe.IntegrityList object at 0x03F50F88>]'. Reason: 'TypeError("cannot pickle '_thread.RLock' object")'

Here is what my code looks like:

from multiprocessing import Pool

p = Pool()
ilList = p.starmap(extract_excel, [(f, spaltennamen) for f in files])
p.join()
p.close()

And this is the function I am trying to execute parallely:

def extract_excel(t: tuple) -> IntegrityList:

file_path = t[0]
spaltennamen = t[1]

il = IntegrityList(file_path)
print(il)

spaltennamen = list(map(lambda x: Excel.HereOrFind(il.ws, x, x.value), spaltennamen))  # Update position of column headers

il.parse_columns(spaltennamen, il.ws)

il.close()

return il

Since I am quite new to python, I am having troubles figuring out the magic behind this multiprocessing error. Executing the function serially is working perfectly fine and I get the desired output. This proofs, that the function and all the sub functions work as expected. I would be glad for any information that could help solve this problem. Thanks!

CodeSocke
  • 91
  • 1
  • 7

1 Answers1

2

Okay so for future issue viewers, I solved the error with the help of this website: https://www.synopsys.com/blogs/software-security/python-pickling/#:~:text=Whenever%20an%20object%20is%20pickled,reconstruct%20this%20object%20when%20unpickling..

It states that every customized object that goes through a parallel process needs to have the __reduce__ method implemented in order to be reconstructed.

I simply added this code to my custom object:

def __reduce__(self):
    return IntegrityList, (self.file_path,)

After that the execution of the parallel processing works great.

CodeSocke
  • 91
  • 1
  • 7