2

I have this simple example of map using multiprocessing. But even this I can't make run correctly.

import multiprocessing

p = multiprocessing.Pool()

rere = range(50)
print p.map(lambda x: x+1, rere)

It will print this exception:

Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 808, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 761, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 342, in _handle_tasks
    put(task)
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

And be unable to terminate with Ctrl+C.

How should I fix my example to make it work?

LtWorf
  • 7,286
  • 6
  • 31
  • 45
  • yea, i jumped the gun sorry. it has to do with the way multiprocessing.pool.map works, the part `lambda x: x+1` has to be a pickleable, not a function, more experimentation is required by both of us! – TehTris May 21 '13 at 16:08
  • 2
    http://stackoverflow.com/questions/8804830/python-multiprocessing-pickling-error?rq=1 check this out, its almost the exact same – TehTris May 21 '13 at 16:13
  • Declaring the function as top level in the module i get this: AttributeError: 'module' object has no attribute 'f' (of course my function is called f, and it claims it doesn't exist for some reason). – LtWorf May 21 '13 at 16:42

1 Answers1

1

OK so from my research into the strange world of multiprocessing...

The way you are trying to do it, is not enough. Heres how I managed to pull it off.

import multiprocessing as mp
import time

def theGenerator():
    for number in xrange(10):
        yield number

def processNumber(x):
    return x*2

def multiprocessThings():
    pool = mp.Pool()
    gen = theGenerator()
    result = pool.map(processNumber, gen)
    print result

if __name__ == "__main__":
    multiprocessThings()
    time.sleep(10)

Save it wherever, and then double click on it.

[0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

also apparently this type of stuff will NOT work in the interpreter for some reason.

TehTris
  • 3,139
  • 1
  • 21
  • 33
  • Thanks, this one works! (and i don't double click, i use the command line). – LtWorf May 21 '13 at 19:28
  • No problem! I did learning here too ^_^. Its sorta barebones-ish so modifying it to whatever you want shouldnt be too much more work. – TehTris May 21 '13 at 19:37
  • Another thing i just found out.... VERY IMPORTANT to make sure that whatever your function is ( in my example its `processNumber()` ) make sure that it is TOP level! for some reason that is very important. – TehTris May 22 '13 at 17:06