0

I have a piece of code stored in a string (user edits this in a text box in my application). In this code, there is a function which I want to run later using multiprocess.Pool. But it does not work if the functions defined in the string are nested.

Note that I am using multiprocess, rather than standard multiprocessing, which would not be able to run the functions defined in the string at all.

See the comments in the following snippet to see what happens. Is there any workaround which I could use in my scenario?

funcs = """
def func(x):
    return func_inner(x) + 1

def func_inner(x):
    return x + 1
"""

import multiprocess

if __name__ == "__main__":
    symbols = dict()
    exec(funcs, symbols)
    func = symbols["func"]
    print(func(1))  # this is evaluated correctly as 3 so nesting functions work fine!
    data = [1, 2, 3]
    pool = multiprocess.Pool()
    print(pool.map(func, data))  # this fails with NameError: name 'func_inner' is not defined!

I tried this on Windows and Linux as well. My multiprocess version is 0.70.5 and Python 3.5.

HiFile.app - best file manager
  • 7,273
  • 2
  • 32
  • 80

1 Answers1

0

Seems you're just missing compile() for your string based function:

funcs=compile("""
def func(x):
    return func_inner(x) + 1

def func_inner(x):
    return x + 1
""", '<string>', 'exec')

if __name__ == "__main__":
   import multiprocess

   symbols = dict()
   exec(funcs, symbols)
   func = symbols["func"]
   print(func(1))
   data = [1, 2, 3]
   pool = multiprocess.Pool()
   print(pool.map(func, data))

outputs for me: [3, 4, 5]

UPDATE: my multiprocess version is - 0.70.4 try the following dirty fix, should work for you:

import multiprocess

code_globals = {}
code_locals = {}
exec funcs in code_globals, code_locals
print code_locals
# {'func_inner': <function func_inner at 0x10aa607d0>, 'func': <function func at 0x10a97dde8>}

code_globals['func_inner']=code_locals['func_inner']

print code_globals['func_inner']
#<function func_inner at 0x10a1427d0>

func = code_locals['func']
print(func(1))
data = [1, 2, 3]
pool = multiprocess.Pool()
print(pool.map(func, data))
brc
  • 249
  • 2
  • 6
  • I am afraid it still fails for me with the same error as before. – HiFile.app - best file manager May 22 '17 at 08:24
  • Which `multiprocess` version do you have? I suspect that certain things got broken in recent versions. – HiFile.app - best file manager May 22 '17 at 08:34
  • I can confirm that after adding `code_globals['func_inner']=code_locals['func_inner']`, the code snippet works. But the problem is that in real life this cannt be used because I do not know in advance, which function will be called by the pool, which is nested etc. I would need something like `code_globals.update(code_locals)` which would copy all local symbols to global symbols. But with this line, the sample gets broken again. – HiFile.app - best file manager May 22 '17 at 09:22
  • Btw. I also tried to install older version of `multiprocess` which you mention - 0.70.4. Still the same problem. There must be something else which make it broken. – HiFile.app - best file manager May 22 '17 at 09:33
  • what python version do you use? mine is 2.7.13. Also this woks for me just fine: 'code_globals.update(code_locals)'. Try to print out 'code_locals' and 'code_globals' before and after update to compare – brc May 22 '17 at 09:35
  • I am afraid my Python 2.7.12 seems broken beyond repair. It does not even print correctly `print func(1)`. :( – HiFile.app - best file manager May 22 '17 at 10:30
  • Moreover, even when I manage to run the example with hacking `globals` and `locals` (by `update` and then deleting the main function from it) I always get into troubles when trying to add one more level of nesting or recursive function calls. – HiFile.app - best file manager May 22 '17 at 10:35