multiprocessing.Pool - PicklingError: Can't pickle : attribute lookup thread.lock failed

Question

multiprocessing.Pool is driving me crazy...
I want to upgrade many packages, and for every one of them I have to check whether there is a greater version or not. This is done by the check_one function.
The main code is in the Updater.update method: there I create the Pool object and call the map() method.

Here is the code:

def check_one(args):
    res, total, package, version = args
    i = res.qsize()
    logger.info('\r[{0:.1%} - {1}, {2} / {3}]',
        i / float(total), package, i, total, addn=False)
    try:
        json = PyPIJson(package).retrieve()
        new_version = Version(json['info']['version'])
    except Exception as e:
        logger.error('Error: Failed to fetch data for {0} ({1})', package, e)
        return
    if new_version > version:
        res.put_nowait((package, version, new_version, json))

class Updater(FileManager):

    # __init__ and other methods...

    def update(self):    
        logger.info('Searching for updates')
        packages = Queue.Queue()
        data = ((packages, self.set_len, dist.project_name, Version(dist.version)) \
            for dist in self.working_set)
        pool = multiprocessing.Pool()
        pool.map(check_one, data)
        pool.close()
        pool.join()
        while True:
            try:
                package, version, new_version, json = packages.get_nowait()
            except Queue.Empty:
                break
            txt = 'A new release is avaiable for {0}: {1!s} (old {2}), update'.format(package,
                                                                                      new_version,
                                                                                      version)
            u = logger.ask(txt, bool=('upgrade version', 'keep working version'), dont_ask=self.yes)
            if u:
                self.upgrade(package, json, new_version)
            else:
                logger.info('{0} has not been upgraded', package)
        self._clean()
        logger.success('Updating finished successfully')

When I run it I get this weird error:

Searching for updates
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/multiprocessing/pool.py", line 225, in _handle_tasks
    put(task)
PicklingError: Can't pickle <type 'thread.lock'>: attribute lookup thread.lock failed

unutbu · Accepted Answer · 2019-06-15T10:36:10.857

multiprocessing passes tasks (which include check_one and data) to the worker processes through a mp.SimpleQueue. Unlike Queue.Queues, everything put in the mp.SimpleQueue must be pickable. Queue.Queues are not pickable:

import multiprocessing as mp
import Queue

def foo(queue):
    pass

pool=mp.Pool()
q=Queue.Queue()

pool.map(foo,(q,))

yields this exception:

UnpickleableError: Cannot pickle <type 'thread.lock'> objects

Your data includes packages, which is a Queue.Queue. That might be the source of the problem.

Here is a possible workaround: The Queue is being used for two purposes:

to find out the approximate size (by calling qsize)
to store results for later retrieval.

Instead of calling qsize, to share a value between multiple processes, we could use a mp.Value.

Instead of storing results in a queue, we can (and should) just return values from calls to check_one. The pool.map collects the results in a queue of its own making, and returns the results as the return value of pool.map.

For example:

import multiprocessing as mp
import Queue
import random
import logging

# logger=mp.log_to_stderr(logging.DEBUG)
logger = logging.getLogger(__name__)


qsize = mp.Value('i', 1)
def check_one(args):
    total, package, version = args
    i = qsize.value
    logger.info('\r[{0:.1%} - {1}, {2} / {3}]'.format(
        i / float(total), package, i, total))
    new_version = random.randrange(0,100)
    qsize.value += 1
    if new_version > version:
        return (package, version, new_version, None)
    else:
        return None

def update():    
    logger.info('Searching for updates')
    set_len=10
    data = ( (set_len, 'project-{0}'.format(i), random.randrange(0,100))
             for i in range(set_len) )
    pool = mp.Pool()
    results = pool.map(check_one, data)
    pool.close()
    pool.join()
    for result in results:
        if result is None: continue
        package, version, new_version, json = result
        txt = 'A new release is avaiable for {0}: {1!s} (old {2}), update'.format(
            package, new_version, version)
        logger.info(txt)
    logger.info('Updating finished successfully')

if __name__=='__main__':
    logging.basicConfig(level=logging.DEBUG)
    update()

Thank you! But now how can I populate my queue? I cannot make it global... I also tried to make `check_one` a method (with this hack: http://stackoverflow.com/questions/1816958/cant-pickle-type-instancemethod-when-using-pythons-multiprocessing-pool-ma), but again, it didn't work... — rubik, Oct 23 '11 at 10:43
I found the solution here: http://stackoverflow.com/questions/3217002/how-do-you-pass-a-queue-reference-to-a-function-managed-by-pool-map-async. I have to use `multiprocessing.Manager().Queue()` instead of `multiprocessing.Queue`. — rubik, Oct 23 '11 at 11:06
As for the workarounds, I think you're right. I'll fix my code. Thank you again!! — rubik, Oct 23 '11 at 11:14
I know this question addresses `multiprocessing.Pool`, but what do I do if I have an actual `threading.Lock`? — Jeppe, Aug 10 '20 at 08:15

score 11 · Answer 2 · answered Jul 29 '16 at 03:28

11

After a lot of digging on a similar issue...

It also turns out that ANY object that happens to contain a threading.Condition() object will NEVER NEVER work with multiprocessing.Pool.

Here is an example

import multiprocessing as mp
import threading

class MyClass(object):
   def __init__(self):
      self.cond = threading.Condition()

def foo(mc):
   pass

pool=mp.Pool()
mc=MyClass()
pool.map(foo,(mc,))

I'm ran this with Python 2.7.5 and hit the same error:

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
  File "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 342, in _handle_tasks
put(task)
PicklingError: Can't pickle <type 'thread.lock'>: attribute lookup thread.lock failed

But then ran it on python 3.4.1 and this issue has been fixed.

Although I haven't come across any useful workarounds yet for those of us still on 2.7.x.

answered Jul 29 '16 at 03:28

jersey bean

3,321
4
28
43

1

As of Python 3.4.3, `threading.Lock` still cannot be pickled (though as you mentioned, `threading.Condition` can be). – caleb Oct 02 '16 at 02:10
14

I confirm that `threading.Lock` objects still cannot be pickled in Python 3.6.2. – astrojuanlu Dec 07 '17 at 09:06
17

hmm...and still on 3.7 – jersey bean Jul 06 '18 at 02:03
1

version 3.6.8 and having this problem right now, digging stack overflow... – RonanFelipe Aug 14 '19 at 12:59
i had the same issue when i tried to pass a threading.RLock to the Pool initializer. After I switch to the multiprocessing.RLock, everything worked as expected. – Frank Liu Feb 27 '20 at 10:44
@caleb @astrojuanlu Confirm @Frank solution of using `multiprocessing.Lock` instead of `threading.Lock` is picklable and works properly with pools. – Chris Collett May 03 '21 at 19:55

score 1 · Answer 3 · answered Jun 14 '19 at 06:48

1

I experienced this issue with python version 3.6 on docker. Changed the version to 3.7.3 and it was solved.

answered Jun 14 '19 at 06:48

Akalanka Weerasooriya

504
5
10

Facing the issue in Python 3.7.6 – datamansahil Oct 20 '20 at 15:01
Python 3.8 also still struggling – Frankie Drake Apr 25 '22 at 10:49
Python 3.9.13 for me – kev Sep 20 '22 at 13:05
3.11 for me, still face issue – Haiping Fan Feb 08 '23 at 09:05

multiprocessing.Pool - PicklingError: Can't pickle : attribute lookup thread.lock failed

3 Answers3

Linked