multiprocess in python: cross process to process watch-kill-restart

Question

Want to make a "cross-life" control on two processes started by the same parent.

If process A take too long to change a value on the Read list of process B, process B should kill and restart A, and reverse also is true.

This is how process are launched from parent:

def check():
    ....# ok check the value in the R list
    #kill and restart

def A:
    check()#kill and restart B if something goes wrong
    #... do his homework


def B:
    check()#kill and restart A if something goes wrong
    #... do his homework

job1 = multiprocessing.Process(target=A, args=(R,W))
job2 = multiprocessing.Process(target=B, args=(W,R))


procs = []
procs.append(job1)
procs.append(job2)



if __name__ == '__main__':

    _start = time.time()
    map(lambda x: x.start(), procs)                                
    #job1.terminate() it kill the process but can't call it from inside A or B
    map(lambda x: x.join(), procs)

Both A and B have a call (and this work) to a function that is defined in the parent and look for the changes in the list but how can I call the terminate() and run() from that function that run inside of the processes so can't know the name, pid of the other process?

Are you sure you wanna do this? This is breaking rules of safe and reliable concurrent programming. If you launch two processes from _supervisor_ use it as a mediator between A and B. You can create something like *watchdog* or send message from A to _supervisor_, that B don't reply. — Michał Fita, Jun 17 '13 at 10:19
Supervisor seems to be interesting but it seems to be unable to run under python 3, I need to build an application without use of external source as far as I can. About the watchdog you mean to write new cod that send a special value in the exchange list to trigger the process kill from inside? or there is already watchdog function in multiprocessing? — user2239318, Jun 17 '13 at 12:30
I mean to code something on your own, as `multiprocessing` has mechanics for inter-process communication. — Michał Fita, Jun 17 '13 at 13:05
ok, I don't ask the code, I'm asking if someone with better experience than me can suggest "a best practice" steps to do it. Mean is better that the watch is handled inside A and B or by the parent? or another process that only do the check? — user2239318, Jun 17 '13 at 13:53
By parent. As probably parent will no nothing else than waiting for those to to finish the job. — Michał Fita, Jun 17 '13 at 13:59
I've turned A and B in classes and added a shutdownrestart method that kill the failing process but after this if I try A.start() or A.run() this error appear 'cannot start a process twice'. — user2239318, Jun 25 '13 at 10:01

score 0 · Answer 1 · answered Jul 26 '13 at 12:00

If I kill a process that share a list with multiprocessing.Manager is there a way to i.e. set all values to "0" and prevent the crash of others programs that will look for the same shared list?

This is the output in console if I kill the process that share the list and then restart it:

parent process: 19416
process id: 19500
[DEBUG/gigi-5:20] thread 'MainThread' does not own a connection
[DEBUG/gigi-5:20] making connection to manager
[DEBUG/gigi-5:20] failed to connect to address /tmp/pymp-0HGU9Z/listener-HVxKEA
[INFO/gigi-5:20] process shutting down
[DEBUG/gigi-5:20] running all "atexit" finalizers with priority >= 0
[DEBUG/gigi-5:20] running the remaining "atexit" finalizers
Process Pippo-5:20:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/gigi/TESTCOM/gigi.py", line 62, in run
    COM.pid2.value=COM.info('gigi')                                  # assegna a pid1 il PID del processo
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 1040, in set
    return self._callmethod('set', (value,))
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 755, in _callmethod
    self._connect()
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 742, in _connect
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "/usr/lib/python2.7/multiprocessing/connection.py", line 169, in Client
    c = SocketClient(address)
  File "/usr/lib/python2.7/multiprocessing/connection.py", line 297, in SocketClient
    s.connect(address)
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 2] No such file or directory
[INFO/gigi-5:20] process exiting with exitcode 1

score -1 · Answer 2 · answered Jun 25 '13 at 13:36

I've enabled Debug functions, and tried to kill and restart child processes from the parent.

I've turned A and B in classes and added a shutdown method that kill the failing process, after this, if I try A.start() or A.run() this error appear 'cannot start a process twice'.What this means? the process is dead, there aren't others running.

If I call multiprocessing.active_childre() the process aren't restarted but got this output:

 <Process(SyncManager-2, started)>, <Process(SyncManager-1, started)>, <Process(SyncManager-3, started)>]
[INFO/MainProcess] process shutting down
[DEBUG/MainProcess] running all "atexit" finalizers with priority >= 0
[SUBDEBUG/MainProcess] calling <Finalize object, callback=_decref, args=(Token(typeid='Value', address='/tmp/pymp-2ObnP1/listener-aIRbnG', id='7f28db9b6190'), '\xc0\xad;K\xce2\xb1\x0f\x17\xe4|f\x14\x1a\x11\xc0\x89 n\x00\x00.\x19B0\xcd%\xac \x83\x8e\x87', <multiprocessing.managers.State object at 0x7f28db9a6718>, <multiprocessing.util.ForkAwareLocal object at 0x7f28db98da10>, ProcessLocalSet(['7f28db9b6190']), <function Client at 0x7f28db99b9b0>), exitprority=10>
[SUBDEBUG/MainProcess] finalizer calling <function _decref at 0x7f28db99faa0> with args (Token(typeid='Value', address='/tmp/pymp-2ObnP1/listener-aIRbnG', id='7f28db9b6190'), '\xc0\xad;K\xce2\xb1\x0f\x17\xe4|f\x14\x1a\x11\xc0\x89 n\x00\x00.\x19B0\xcd%\xac \x83\x8e\x87', <multiprocessing.managers.State object at 0x7f28db9a6718>, <multiprocessing.util.ForkAwareLocal object at 0x7f28db98da10>, ProcessLocalSet(['7f28db9b6190']), <function Client at 0x7f28db99b9b0>) and kwargs {}
[DEBUG/MainProcess] DECREF '7f28db9b6190'
[DEBUG/MainProcess] thread 'Dummy-1' has no more proxies so closing conn
[SUBDEBUG/MainProcess] calling <Finalize object, callback=_decref, args=(Token(typeid='Value', address='/tmp/pymp-EfUPHj/listener-hPdyzk', id='7f28db9b30d0'), '\xc0\xad;K\xce2\xb1\x0f\x17\xe4|f\x14\x1a\x11\xc0\x89 n\x00\x00.\x19B0\xcd%\xac \x83\x8e\x87', <multiprocessing.managers.State object at 0x7f28db9a6638>, <multiprocessing.util.ForkAwareLocal object at 0x7f28db98d9a8>, ProcessLocalSet(['7f28db9b30d0']), <function Client at 0x7f28db99b9b0>), exitprority=10>
[SUBDEBUG/MainProcess] finalizer calling <function _decref at 0x7f28db99faa0> with args (Token(typeid='Value', address='/tmp/pymp-EfUPHj/listener-hPdyzk', id='7f28db9b30d0'), '\xc0\xad;K\xce2\xb1\x0f\x17\xe4|f\x14\x1a\x11\xc0\x89 n\x00\x00.\x19B0\xcd%\xac \x83\x8e\x87', <multiprocessing.managers.State object at 0x7f28db9a6638>, <multiprocessing.util.ForkAwareLocal object at 0x7f28db98d9a8>, ProcessLocalSet(['7f28db9b30d0']), <function Client at 0x7f28db99b9b0>) and kwargs {}
[DEBUG/MainProcess] DECREF '7f28db9b30d0'
[DEBUG/MainProcess] thread 'Dummy-1' has no more proxies so closing conn
[SUBDEBUG/MainProcess] calling <Finalize object, callback=_decref, args=(Token(typeid='list', address='/tmp/pymp-W6FZ5G/listener-V4fUuw', id='7f28db9b3908'), '\xc0\xad;K\xce2\xb1\x0f\x17\xe4|f\x14\x1a\x11\xc0\x89 n\x00\x00.\x19B0\xcd%\xac \x83\x8e\x87', <multiprocessing.managers.State object at 0x7f28ddc2f7f8>, <multiprocessing.util.ForkAwareLocal object at 0x7f28db98d940>, ProcessLocalSet(['7f28db9acf38', '7f28db9b3908']), <function Client at 0x7f28db99b9b0>), exitprority=10>
[SUBDEBUG/MainProcess] finalizer calling <function _decref at 0x7f28db99faa0> with args (Token(typeid='list', address='/tmp/pymp-W6FZ5G/listener-V4fUuw', id='7f28db9b3908'), '\xc0\xad;K\xce2\xb1\x0f\x17\xe4|f\x14\x1a\x11\xc0\x89 n\x00\x00.\x19B0\xcd%\xac \x83\x8e\x87', <multiprocessing.managers.State object at 0x7f28ddc2f7f8>, <multiprocessing.util.ForkAwareLocal object at 0x7f28db98d940>, ProcessLocalSet(['7f28db9acf38', '7f28db9b3908']), <function Client at 0x7f28db99b9b0>) and kwargs {}
[DEBUG/MainProcess] DECREF '7f28db9b3908'
[SUBDEBUG/MainProcess] calling <Finalize object, callback=_decref, args=(Token(typeid='list', address='/tmp/pymp-W6FZ5G/listener-V4fUuw', id='7f28db9acf38'), '\xc0\xad;K\xce2\xb1\x0f\x17\xe4|f\x14\x1a\x11\xc0\x89 n\x00\x00.\x19B0\xcd%\xac \x83\x8e\x87', <multiprocessing.managers.State object at 0x7f28ddc2f7f8>, <multiprocessing.util.ForkAwareLocal object at 0x7f28db98d940>, ProcessLocalSet(['7f28db9acf38']), <function Client at 0x7f28db99b9b0>), exitprority=10>
[SUBDEBUG/MainProcess] finalizer calling <function _decref at 0x7f28db99faa0> with args (Token(typeid='list', address='/tmp/pymp-W6FZ5G/listener-V4fUuw', id='7f28db9acf38'), '\xc0\xad;K\xce2\xb1\x0f\x17\xe4|f\x14\x1a\x11\xc0\x89 n\x00\x00.\x19B0\xcd%\xac \x83\x8e\x87', <multiprocessing.managers.State object at 0x7f28ddc2f7f8>, <multiprocessing.util.ForkAwareLocal object at 0x7f28db98d940>, ProcessLocalSet(['7f28db9acf38']), <function Client at 0x7f28db99b9b0>) and kwargs {}
[DEBUG/MainProcess] DECREF '7f28db9acf38'
[DEBUG/MainProcess] thread 'Dummy-1' has no more proxies so closing conn
[SUBDEBUG/MainProcess] calling <Finalize object, callback=_finalize_manager, args=(<Process(SyncManager-3, started)>, '/tmp/pymp-2ObnP1/listener-aIRbnG', '\xc0\xad;K\xce2\xb1\x0f\x17\xe4|f\x14\x1a\x11\xc0\x89 n\x00\x00.\x19B0\xcd%\xac \x83\x8e\x87', <multiprocessing.managers.State object at 0x7f28db9a6718>, <function Client at 0x7f28db99b9b0>), exitprority=0>
[SUBDEBUG/MainProcess] finalizer calling <function _finalize_manager at 0x7f28db99f5f0> with args (<Process(SyncManager-3, started)>, '/tmp/pymp-2ObnP1/listener-aIRbnG', '\xc0\xad;K\xce2\xb1\x0f\x17\xe4|f\x14\x1a\x11\xc0\x89 n\x00\x00.\x19B0\xcd%\xac \x83\x8e\x87', <multiprocessing.managers.State object at 0x7f28db9a6718>, <function Client at 0x7f28db99b9b0>) and kwargs {}
[INFO/MainProcess] sending shutdown message to manager
[SUBDEBUG/MainProcess] calling <Finalize object, callback=_finalize_manager, args=(<Process(SyncManager-2, started)>, '/tmp/pymp-EfUPHj/listener-hPdyzk', '\xc0\xad;K\xce2\xb1\x0f\x17\xe4|f\x14\x1a\x11\xc0\x89 n\x00\x00.\x19B0\xcd%\xac \x83\x8e\x87', <multiprocessing.managers.State object at 0x7f28db9a6638>, <function Client at 0x7f28db99b9b0>), exitprority=0>
[SUBDEBUG/MainProcess] finalizer calling <function _finalize_manager at 0x7f28db99f5f0> with args (<Process(SyncManager-2, started)>, '/tmp/pymp-EfUPHj/listener-hPdyzk', '\xc0\xad;K\xce2\xb1\x0f\x17\xe4|f\x14\x1a\x11\xc0\x89 n\x00\x00.\x19B0\xcd%\xac \x83\x8e\x87', <multiprocessing.managers.State object at 0x7f28db9a6638>, <function Client at 0x7f28db99b9b0>) and kwargs {}
[INFO/MainProcess] sending shutdown message to manager
[SUBDEBUG/MainProcess] calling <Finalize object, callback=_finalize_manager, args=(<Process(SyncManager-1, started)>, '/tmp/pymp-W6FZ5G/listener-V4fUuw', '\xc0\xad;K\xce2\xb1\x0f\x17\xe4|f\x14\x1a\x11\xc0\x89 n\x00\x00.\x19B0\xcd%\xac \x83\x8e\x87', <multiprocessing.managers.State object at 0x7f28ddc2f7f8>, <function Client at 0x7f28db99b9b0>), exitprority=0>
[SUBDEBUG/MainProcess] finalizer calling <function _finalize_manager at 0x7f28db99f5f0> with args (<Process(SyncManager-1, started)>, '/tmp/pymp-W6FZ5G/listener-V4fUuw', '\xc0\xad;K\xce2\xb1\x0f\x17\xe4|f\x14\x1a\x11\xc0\x89 n\x00\x00.\x19B0\xcd%\xac \x83\x8e\x87', <multiprocessing.managers.State object at 0x7f28ddc2f7f8>, <function Client at 0x7f28db99b9b0>) and kwargs {}
[INFO/MainProcess] sending shutdown message to manager
[DEBUG/MainProcess] running the remaining "atexit" finalizers

multiprocess in python: cross process to process watch-kill-restart

2 Answers2