0

I'm trying to create a series of worker threads that have specific jobs. The current is to draw from a queue, do some processing, and put the output in a second queue.

When I try to use the multiprocessing (multiprocess? what's the difference?) module I get the following issue

Traceback (most recent call last):
  File "C:\Python27\lib\multiprocessing\queues.py", line 264, in _feed
    send(obj)
PicklingError: Can't pickle <type 'function'>: attribute lookup     __builtin__.function failed

It seems to be a cpickle issue of some kind. I am processing objects with __getattr__ overwritten but I did use

if name.startswith('__') and name.endswith('__'):
        return object.__getattr__(name)

to ensure that the TypeError is not called anymore. The answer to pickling issues with multiprocessing is to use pathos but I don't see a way to access any kind of queues in pathos. I also don't see a way to spawn a Process in pathos. Every example I've seen uses a pool. I also saw a response that there is pathos.helpers.mp.process.Process but the modules cannot be found for me - ImportError: cannot import name helpers, for example. Is this something possible?

Ryan
  • 80
  • 10

1 Answers1

1

I'm the pathos author. Yes, you can use a Process from pathos, however, if that is what you are interested in, it's probably easier to just use multiprocess. Some disambiguation: multiprocess is a fork of multiprocessing, where the fork replaces pickle with dill -- and there's no other changes. pathos, then adds an additional layer on top of multiprocess, primarily on the Pool objects. pathos.multiprocessing is the additional layer on top of multiprocess… and if you want direct access to multiprocess from pathos, you can get it. See:

>>> import pathos
>>> pathos.helpers.mp.Process
<class 'multiprocess.process.Process'>
>>> 
>>> import multiprocess as mp
>>> mp.Process
<class 'multiprocess.process.Process'>
>>>
Mike McKerns
  • 33,715
  • 8
  • 119
  • 139
  • Thanks for the reply. I get the following issue exception when I try that `AtributeError: 'module' object has no attribute 'helpers'`. I tried playing with some things and it seems that another huge issue I'm having is that you cannot share `lxml` instances with different processes, even if the object itself can `pickle`. – Ryan Mar 14 '16 at 12:29
  • Do you have a recent version of `pathos` from github? (i.e. you didn't install the pathos release, you installed `pathos` master from github) – Mike McKerns Mar 15 '16 at 00:29
  • currently its moved to: import pathos.multiprocessing as mpm; mpm._ProcessPool.Process – yourstruly May 01 '16 at 19:34
  • @RobertGrzelka: It's not moved. It's found at both locations. – Mike McKerns May 02 '16 at 01:54
  • xD I was looking for it using: help(pathos.multiprocessing); and there it was at the end of text... It would be nice to have it accesible from pathos.multiprocessing directly :P – yourstruly May 02 '16 at 07:46
  • @RobertGrzelka: yes, maybe some reorganization would be useful – Mike McKerns May 02 '16 at 12:42