0

I think my question was not very clear before.. I'm trying to create a class module that includes a function does a mutiprocessing of a ctpyes function.

I'm re-posting a working small code. All I want to do is to remove the code below and call the function directly from my class. But it seems very hard since ctypes object is not picklable.. Is there a way to resolve this issue? Even a small hint will be appreciated!! Will I have to switch to using cython instead of ctypes? will that even help?

def myc(x):
    return a.funct(x)

Below is the working code.

from ctypes import *
from ctypes.util import find_library
import multiprocess as mp
class ctest():
    def __init__(self):
        self.libapr = cdll.LoadLibrary(find_library('apr-1'))
        self.libapr.apr_fnmatch.argtypes = [c_char_p, c_char_p, c_int]
        self.libapr.apr_fnmatch.restype = c_int
    def funct(self,x):
        y=self.libapr.apr_fnmatch(x, 'name.ext', 0)
        return y
    def mymult(self,func,xlist):       
        pool=mp.Pool(20)
        res=pool.map(func,xlist)
        pool.close()
        return res
if __name__ == "__main__":
    a=ctest()
    def myc(x):return a.funct(x)
    print a.mymult(myc,['*.txt','*.ext'])

Below is what I want to do.

from ctypes import *
from ctypes.util import find_library
import multiprocess as mp
class ctest():
    def __init__(self):
        self.libapr = cdll.LoadLibrary(find_library('apr-1'))
        self.libapr.apr_fnmatch.argtypes = [c_char_p, c_char_p, c_int]
        self.libapr.apr_fnmatch.restype = c_int
    def funct(self,x):
        y=self.libapr.apr_fnmatch(x, 'name.ext', 0)
        return y
    def mymult(self,func,xlist):       
        pool=mp.Pool(20)
        res=pool.map(func,xlist)
        pool.close()
        return res
if __name__ == "__main__":
    a=ctest()
    a.mymult(a.funct,['*.txt','*.ext'])
윤제균
  • 149
  • 1
  • 10
  • Hi, I'm the author of `multiprocess` and `dill`. What is your question? Is the question: "Can you make my code work?" or "What's the best approach?"... Python will have a lot of trouble pickling `"C"` objects, so yes, handling them in parallel can be tricky. – Mike McKerns Jul 29 '18 at 11:08
  • Hi, thanks fo the reply! Maybe my question was not very clear. My question is basically, if there is a way to put the multiprocessing part inside my class module. – 윤제균 Jul 29 '18 at 14:28
  • Two things: (1) isn't it inside the class already? (i.e. in the `mymult` method), and (2) typically, it's not a good idea to put a pool and map inside a class -- it's much better to expect a pool or a map, and then pass an instance in as an argument... or even more simply, just use an external pool/map. – Mike McKerns Jul 29 '18 at 23:18
  • (1)... Technically, it is not... if I want to distribute this module to other users, the user will have to define extra function each time they want to use the multi processing.. I'm designing this module to be linked with a GUI platform later on but I think, in this way it will make the GUI to call the multiprocessing very hard.... Do you know if there is any other ways to parallelise my function call? Using the multiprocessing reduces the entire simulation time up to 10x... – 윤제균 Jul 30 '18 at 02:15
  • I'm not sure that your edit, or comment above has clarified the situation. You already have the `Pool` and `map` inside a method (i.e. my (1)). Are you saying that you also want to have the target of the map to be a method of the class (i.e. `myc` becomes a class method)? With regard to your edits, no... going to cython should not make things more serializable. Success in serialization depends on the pointer hierarchy of the objects that you are intending to pickle. – Mike McKerns Jul 30 '18 at 10:58
  • Mike, many thanks for your previous comment! I found workaround with my previous code and survived for a year but now I'm facing a big issue with the my code structure. I've edited and added the code structure in a way I want.. Basically, I want to excute "a=ctest() a.mymult(a.funct,['*.txt','*.ext'])" this part inside another class or method. Sorry for my ignorance if it is too obvious problem. – 윤제균 Oct 16 '19 at 07:48
  • Maybe the best thing to do is ask a new question and link it back to this question. It's difficult to discern what you want to do within the compact nature of the comments. – Mike McKerns Oct 16 '19 at 10:37

0 Answers0