Show the running executors identifications in ProcessPoolExecutor

Question

This is the problem I am having. I wont share code because of condfidentiality but instead I will provide some dummy example.

Assume that we have a class as follows:

class SayHello:
   def __init__(self, name, id):
      self.name=name
      self.id=id
   
   #public func
   def doSomething(self, arg1, arg2 ):
      DoAHugeTaskWithArgument

Lets say now that in an other modules we have this:

class CallOperations:
    def __init__(self):
       self.dummydict={1: {"james":20, "peter":30, "victor":40, "john":45, "ali":21, "tom":41, "hector":37}, 2:{"james":23, "peter":31, "victor":44, "john":46, "ali":23, "tom":44, "hector":35} }
    
    def runProcessors(self):
       #runprocess
       
       for _, v in self.dummydict.items():
          Instances = [SayHello(g,b) for g ,b in v.items()] 
          with ProcessPoolExecutor(max_workers=2) as executor: 
               future = [executor.submit(ins.doSomething, 2, 1235) for ins in Instances]

So the problem starts here. I want to know what instances are running doSomething() funtion in their respective process. I want to set a variable = 1 when the function of that instance is running in the process and set it to zero when it is completed.

Each instance has its own name and id. Is there way to find out the name of the running instance in the process?

This problem is making me very confused and can not find a proper solution.

Thank you alot.

Do I understand your question correctly - you want to know when a certain instance of `SayHello` is executing and when it is not? — Hannu, Mar 17 '21 at 15:17

Hannu · Answer 1 · 2021-03-23T17:54:43.160

If I understand your question correctly, you want to know when an instance of SayHello is executing and when it is not. You can set a variable (1 or 0) by using a Manager - but the usefulness of this can be debated. You might want to use a lock instead.

I had to tweak your code a bit but this is a running example. It picks one of your tasks as the one to monitor in the while loop. It is a dummy loop that never exits but you'll get the idea. It will keep polling the variable of one of your instances and you can see it change when that task is running, and then revert back to zero.

from time import sleep
from concurrent.futures import ProcessPoolExecutor
from multiprocessing import Manager

class SayHello:
   def __init__(self, name, id):
      self.name=name
      self.id=id
      self.status = Manager().Value("i",0)

   #public func
   def doSomething(self, arg1, arg2 ):
      self.status.value = 1
      sleep(5)
      self.status.value = 0


class CallOperations:
    def __init__(self):
       self.dummydict={1: {"james":20, "peter":30, "victor":40, "john":45, "ali":21, "tom":41, "hector":37}, 2:{"james":23, "peter":31, "victor":44, "john":46, "ali":23, "tom":44, "hector":35} }

    def runProcessors(self):
       #runprocess

       for _, v in self.dummydict.items():
          Instances = [SayHello(g,b) for g ,b in v.items()]
       f = Instances[3]

       executor = ProcessPoolExecutor(max_workers=2)
       future = [executor.submit(ins.doSomething, 2, 1235) for ins in Instances]
       while True:
          print(f.status.value)
          # Insert break condition here
          sleep(0.5)

       executor.shutdown()

foo = CallOperations()
foo.runProcessors()

The problem with this is that it can lead to a race condition depending on what you do in your main program. If you want to do any operations on the instance when it is in passive state, it might progress to active just after you check the variable but before you have completed your actions in the main program.

Locks come to rescue here, as you can also create a shared lock Manager().Lock(). If your DoSomething() tries to acquire the lock and your main process does the same when operating on a passive instance, you avoid this problem. Of course your main program could then block the executor from processing if it reserves locks for lengthy operations, as then your two workers would be stuck waiting on locks if the execution processed to those instances where locks are being held by the main program. This case would not be suitable for parallel processing implemented using executors.

EDIT: if you are only interested in the running status, you can check the Future.running() status of your future objects, in this case items in your future array.

Thank you for your answer. but why do you have a `f` variable? — Payam30, Mar 23 '21 at 14:23
Thank you for your answer. but why do you have a `f` variable? . Will the Manager be more global? if your code you can see if the fourth instance is running. so by other works I have to loop through all the instance to see if they are running? Can you not get the running status from the future object? @Hannu — Payam30, Mar 23 '21 at 14:33
I only use `f` for matching in the while loop, to demonstrate how this works. You don't need f anywhere. I could have used instances[3].status.value in the while loop, it was just left behind when I piloted this. — Hannu, Mar 23 '21 at 17:52
I have edited my answer. Yes, you can indeed use `running()` method on your future objects (not the array, individual futures so you have to loop through something with this option as well). The problem with `running()` is that it does not allow you to do much with the information except to maybe print a dashboard. If you want to be sure the task does not start running while you either read or write its structures, you cannot use this. Even if the task is not running now, it can start running just after your if statement. — Hannu, Mar 23 '21 at 17:57

Show the running executors identifications in ProcessPoolExecutor

1 Answers1