IPython ipyparallel map_sync ImportError

Question

I am new to ipyparallel and I want to use this package to implement parallel computing with my machine learning application.

The following are a test on ipyparallel, I define a function named add on func.py file, and the main function on test.py file.

The func.py's codes are:

#!/usr/bin/env python
# coding=utf-8

def add(*numbers):
    numbers = list(numbers)
    for i, n in enumerate(numbers):
        numbers[i] = n + 1
    return numbers

The test.py's codes are:

#!/usr/bin/env python
# coding=utf-8

from func import add
from ipyparallel import Client

if __name__ == '__main__':
    rc = Client(
        '/home/fit/.ipython/profile_default/security/ipcontroller-client.json')

    print map(add, [1, 2, 3]
    print rc[0].map_sync(add, [1, 2, 3, 4])

Since you know map can run without error, but when run map_sync, the command line returns:

☁  test  python test.py 
[[2], [3], [4]]
Traceback (most recent call last):
  File "test.py", line 14, in <module>
    print rc[0].map_sync(add, [1, 2, 3, 4])
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/view.py", line 353, in map_sync
    return self.map(f,*sequences,**kwargs)
  File "<string>", line 2, in map
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/view.py", line 54, in sync_results
    ret = f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/view.py", line 618, in map
    return pf.map(*sequences)
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/remotefunction.py", line 268, in map
    ret = self(*sequences)
  File "<string>", line 2, in __call__
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/remotefunction.py", line 75, in sync_view_results
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/remotefunction.py", line 251, in __call__
    return r.get()
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/asyncresult.py", line 104, in get
    raise self._exception
ipyparallel.error.CompositeError: one or more exceptions from call to method: add
[0:apply]: ImportError: No module named func

And if I define the function on the test.py file, map_sync can run:

#!/usr/bin/env python
# coding=utf-8

#from func import add
from ipyparallel import Client

def add(*numbers):
    numbers = list(numbers)
    for i, n in enumerate(numbers):
        numbers[i] = n + 1
    return numbers


if __name__ == '__main__':
    rc = Client(
        '/home/fit/.ipython/profile_default/security/ipcontroller-client.json')

    print map(add, [1, 2, 3])

    print rc[0].map_sync(add, [1, 2, 3, 4])

And the result is:

☁  test  python test.py
[[2], [3], [4]]
[[2], [3], [4], [5]]

I wonder for map_sync how to use the function define in other file? And how to import these functions? Since from py_file import func does not work for map_sync.

GoingMyWay · Accepted Answer · 2015-09-29T07:12:16.880

The module needed should be copied(or, you can push or the modules) to engine machines and 3-party packages should be installed on engine machines, if not, ImportError will be occured.

But, when running the program, you should run:

$ ipcontroller --ip=client_ip --reuse

on client machine, and there will be 2 files generated in /home/user/.ipython/profile_default/security directory

$ ls /home/user/.ipython/profile_default/security 
ipcontroller-client.json  ipcontroller-engine.json

So, both ipcontroller-client.json and ipcontroller-engine.json must be copied to engine machines and run

$ ipengine --file=/path/to/ipcontroller-engine.json

on engine machines, so the parallel computing environment has been set up.

And next, you can define your parallel computing tasks and run your programs.

IPython ipyparallel map_sync ImportError

1 Answers1