7

I have the following setup in Python 3.6 for interprocess communication:

from multiprocessing.managers import BaseManager
class MyManager(BaseManager): pass
MyManager.register('get_instance', callable=lambda:my_instance)
m = MyManager(address=('', 50000), authkey=b'key')
s = m.get_server()
s.serve_forever()

To avoid blocking my message loop of a larger application, I used a thread to contain this setup. So s.serve_forever() actually runs inside a run function of a thread.

This is done according to the documentation. And the setup itself worked fine with client managers calling into the shared instance.

However, I found no way to stop this 'serve_forever' server. Since the documentation didn't mention this, I checked the source code. There's a stop_event which supposedly I can set to quit the loop. But it didn't work, as the accepter daemon/thread is still running. And I can't call shutdown on the server object because I don't have the socket object c.

So how do I shut down this server?

ps. Using BaseManager.start() isn't really an option for me because the server in my case shares an async message loop which can only be accessed by the starting process. BaseManager.start() spawns a new process which no longer has access to the message loop. get_server().serve_forever() on the other hand, runs within the calling process.

He Shiming
  • 5,710
  • 5
  • 38
  • 68
  • The name implies that it will run forever. When do you want to shut it down? When exiting your program, or at a time of your convenience? – JohanL Jul 06 '17 at 08:01
  • Also, have you tried running `serve_forever()` in your main application? It will create a thread to run in automatically. Are you sure it will block your message loop? – JohanL Jul 06 '17 at 09:32

3 Answers3

2

Try this in the server:

import threading

s = m.get_server()
stop_timer = threading.Timer(1, lambda:s.stop_event.set())
MyManager.register('stop', callable=lambda:stop_timer.start())
s.serve_forever()

And in the client:

MyManager.register('stop')
m.stop()

UPDATE:

I solved the timeout issue by delaying the stop_event.set() with a threading.Timer

OLD:

However, across machines, you then run into long timeouts as the stop() method fails to receive data, this answer talks about that - but I could not get it work:

properly disconnect multiprocessing remote manager

I'm new to the multiprocessing managers, I will update this answer if I find a better solution. (UPDATE: Found, updated)

SimonC
  • 21
  • 4
2

Here is a hack after reading the multiprocessing.managers source, stripped from _finalize_manager(), basically creating a connection to the server and send a shutdown msg

from multiprocessing.managers import dispatch,listener_client

_Client = listener_client['pickle'][1]
# address and authkey same as when started the manager
conn = _Client(address=('127.0.0.1', 50000), authkey=b'key')
dispatch(conn, None, 'shutdown')
conn.close()
Flair
  • 2,609
  • 1
  • 29
  • 41
MichaelT
  • 21
  • 1
  • Amazing, thank you so much. I've spent hours on this with no clue why the regular SyncManager.shutdown() function doesn't work. – Aulig Jan 23 '22 at 20:01
1

you can wrap/use the manager start/shutdown to avoid this running forever server (start() starts a server for you):

m = MyManager(address=('', 50000), authkey=b'key')
m.start()
# do something
m.shutdown()
Wajih
  • 905
  • 9
  • 13