Recently, I have been asked to make "our C++ lib work in the cloud". Basically, the lib is computer intensive (calculating prices), so it would make sense. I have constructed a SWIG interface to make a python version with in the mind to use MapReduce with MRJob. I wanted to serialize the objects in a file, and using a mapper, deserialize and calculate the price.
For example:
class MRTest(MRJob):
def mapper(self,key,value):
obj = dill.loads(value)
yield (key, obj.price())
But now I reach a dead end since it seems that dill cannot handle SWIG extension:
PicklingError: Can't pickle <class 'SwigPyObject'>: it's not found as builtins.SwigPyObject
Is there a way to make this work properly?