3

I would like to parallelise a MatLab function using python and ray.

In this example:

  1. I used MathWorks to wrap a very simple matlab function as Python application (the function just returns as result the input integer - see below). Instructions here: https://uk.mathworks.com/help/compiler_sdk/gs/create-a-python-application-with-matlab-code.html

  2. The application (the function) is then called in python (see below). The function is called using the ray package to parallelise the call.

  3. The function is called four times using as inputs some integers: 0, 1, 2 and 3. The result should be a list: [0, 1, 2, 3]

  4. The call fails with error: ray/cloudpickle/cloudpickle_fast.py, TypeError: 'str' object is not callable. The error stays the same whatever the matlab function does and whatever (valid) matlab code I put in.

  5. The same function (test_function_for_python_2) works well in a simple for loop and returns [0, 1, 2, 3], so the problem is most likely not in the MathWorks wrapping of the MatLab function into python.

In summary, ray needs to pickle the matlab/python library (the function), and the pickling encounters a problem I can't solve.

Any help would be most appreciated.

The operating system in this example is MacOSx Catalina, the python version is 3.7.9, the ray python module version is 1.0.0. The mwpython version is 3.7.9 too, and mwpython (installed by the MatLab SDK runtime) must be used to run the python script, not python or python3. MatLab and MatLab SDK runtime are version R2020b.

Matlab function

function result = test_function_for_python_2(an_integer_input)
    result = an_integer_input;
end

Python to parallelises the matlab function above

import TestTwo # a class with a simple function returning the input integer
import matlab
import ray # for parallelisation

# Initialise  ===========
ray.shutdown() # shutdown any open CPU pool 
ray.init() # create a pool of parallel threads 
# initialise the matlab app containing the code of a simple test function (returns the same integer that is input)
my_test_function = TestTwo.initialize() 

print(type(my_test_function))
print(type(my_test_function.test_function_for_python_2))

@ray.remote
def function_two(an_integer):
    '''Here a single integer is input and returned. 
    The function responsible for it has been written in Matlab and wrapped as a python library in Mathworks.'''
    result = my_test_function.test_function_for_python_2(an_integer, 1) # nargout=1)
    return result

# input to the function
some_integers = [0, 1, 2, 3]

# run function and collect results with ray for parallelisation
ray_objects = [function_two.remote(an_integer=i) for i in some_integers] 
future_results = ray.get(ray_objects)

print('Results: {}'.format(future_results))

# Terminate ===== 
my_test_function.terminate() # close the matlab app containing the code for the test function
ray.shutdown() # shutdown any open CPU pool

Error trace from mwpython console

% /Applications/MATLAB/MATLAB_Runtime/v99/bin/mwpython ~/Documents/TestTwo/for_redistribution_files_only/run_test_2_matlab_wrapped_in_python_parallel.py

2020-11-06 21:09:00,142 INFO services.py:1166 -- View the Ray dashboard at http://127.0.0.1:8265
/Applications/MATLAB/MATLAB_Runtime/v99/bin/maci64/mwpython3.7.app/Contents/MacOS/mwpython3.7: can't find '__main__' module in ''

<class 'matlab_pysdk.runtime.deployablepackage.DeployablePackage'>
<class 'matlab_pysdk.runtime.deployablefunc.DeployableFunc'>

Traceback (most recent call last):
  File "/Users/me/Documents/TestTwo/for_redistribution_files_only/run_test_2_matlab_wrapped_in_python_parallel.py", line 32, in <module>
    ray_objects = [function_two.remote(an_integer=i) for i in some_integers] 

  File "/Users/me/Documents/TestTwo/for_redistribution_files_only/run_test_2_matlab_wrapped_in_python_parallel.py", line 32, in <listcomp>
    ray_objects = [function_two.remote(an_integer=i) for i in some_integers] 

  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/ray/remote_function.py", line 99, in _remote_proxy
    return self._remote(args=args, kwargs=kwargs)

  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/ray/remote_function.py", line 207, in _remote
    self._pickled_function = pickle.dumps(self._function)

  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 70, in dumps
    cp.dump(obj)

  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 656, in dump
    return Pickler.dump(self, obj)

TypeError: 'str' object is not callable
user3050287
  • 31
  • 4

0 Answers0