0

I need to hold instances of python objects in different processes, call methods of these objects and collect the results asynchroneously. My approach was to create a manager for each process and object and then to call the repsective methods via the managers:

from multiprocessing.managers import SyncManager

class RemoteObject():
    
    def __init__(self):
        
        print('Start expensive init')
        ... # do expensive stuff
        print('End expensive init')

    def expensiveJob(self, *args, **kwargs):
        print('Start expensive job')
        ... # do expensive stuff
        print('End expensive job')

class MyManager(SyncManager): pass

managers = []
instances = []

numInstances = 4

# Create the remote objects
for i in range(numInstances):
    manager = MyManager()
    manager.register('RemoteObject', RemoteObject)
    manager.start()
    managers.append(manager)
    instances.append(manager.RemoteObject())

# do the jobs
myResult = [instance.expensiveJob() for instance in instances]

Though each of the instances is in a different process now and the jobs are performed in these different processes, both the object initialization and the expensive job happen one after the other. That is, the output is something like

Start expensive init
End expensive init
Start expensive init
End expensive init
Start expensive init
End expensive init
Start expensive init
End expensive init
Start expensive job
End expensive job
Start expensive job
End expensive job
Start expensive job
End expensive job
Start expensive job
End expensive job

whereas I would like to see something like

Start expensive init
Start expensive init
Start expensive init
Start expensive init
End expensive init
End expensive init
End expensive init
End expensive init
Start expensive job
Start expensive job
Start expensive job
Start expensive job
End expensive job
End expensive job
End expensive job
End expensive job

How could I get there? Is there a call_asynchronously method or something like that that I could use? (I have not found something like that in the documentation of the multiprocessing module.)

Note that the classic process pool is not an option here, because I need to hold the remote instances in the memory unchanged between multiple expensive jobs.

Samufi
  • 2,465
  • 3
  • 19
  • 43
  • What OS are you using and how long do the expensive jobs take? I found that on Windows and Mac starting a new process can take several seconds and this happens synchronously so you need big jobs to get benefit. – Simon Notley Oct 24 '20 at 13:23
  • @SimonN I want to start the jobs only once at the beginning of the program and then communicate to the processes through the managers. I am working on Windows but want the program to work on Linux as well. – Samufi Oct 24 '20 at 13:41

0 Answers0