0

I need to attach remote event in distributed machine and processes are many. So I try to append EventProxy in ListProxy for distributed multiprocessing. Codes and error are shown below: Server aims top provide service which is the list.

Client1 aims to append Event in ListProxy and set first Event.

Client2 aims to get Event from ListProxy to figure out whether is set or not

Server:

from multiprocessing.managers import ListProxy,EventProxy
import torch.multiprocessing as mp

def server(host, port, key, manager):
    msg_buffer = manager.list()
    manager.register('get_list', callable=lambda :msg_buffer, proxytype=ListProxy)
    manager.__init__(address=(host, port), authkey=key)
    print('connect to server %s' % host)
    s = manager.get_server()
    s.serve_forever()

if __name__ == '__main__':
    manager = mp.Manager()
    server('127.0.0.1', 5000, b'abc', manager)

Client1:

import torch.multiprocessing as mp

def client1(host, port, key):
    manager = mp.Manager()
    manager.register('get_list')
    manager.__init__(address=(host, port), authkey=key)
    manager.connect()
    return manager

def set_list(manager):
    l = manager.get_list()
    print(l)
    for i in range(3):
        l.append(manager.Event())
    
if __name__ == '__main__':
    manager = client1('127.0.0.1', 5000, b'abc')
    set_list(manager)

Client2:

import torch.multiprocessing as mp


def client1(manager, host, port, key):
    manager.register('get_list')
    manager.__init__(address=(host, port), authkey=key)
    manager.connect()

def set_list():
    l = manager.get_list()
    if l[0].is_set():
        print('set')
    else:
        l[0].set()

if __name__ == '__main__':
    manager = mp.Manager()
    client1(manager, '127.0.0.1', 5000, b'abc')
    set_list()

But got problem shown as below:

Traceback (most recent call last):
  File "/home/drsun/文档/RL/distprocess_prototype/server.py", line 38, in <module>
    server('127.0.0.1', 5000, b'abc', manager)
  File "/home/drsun/文档/RL/distprocess_prototype/server.py", line 25, in server
    event()
  File "/home/drsun/文档/RL/distprocess_prototype/server.py", line 32, in event
    l.append(manager.Event())
  File "<string>", line 2, in append
  File "/home/drsun/anaconda3/envs/AC/lib/python3.6/multiprocessing/managers.py", line 772, in _callmethod
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError: 
---------------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/drsun/anaconda3/envs/AC/lib/python3.6/multiprocessing/managers.py", line 228, in serve_client
    request = recv()
  File "/home/drsun/anaconda3/envs/AC/lib/python3.6/multiprocessing/connection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
  File "/home/drsun/anaconda3/envs/AC/lib/python3.6/multiprocessing/managers.py", line 881, in RebuildProxy
    return func(token, serializer, incref=incref, **kwds)
  File "/home/drsun/anaconda3/envs/AC/lib/python3.6/multiprocessing/managers.py", line 731, in __init__
    self._incref()
  File "/home/drsun/anaconda3/envs/AC/lib/python3.6/multiprocessing/managers.py", line 785, in _incref
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "/home/drsun/anaconda3/envs/AC/lib/python3.6/multiprocessing/connection.py", line 493, in Client
    answer_challenge(c, authkey)
  File "/home/drsun/anaconda3/envs/AC/lib/python3.6/multiprocessing/connection.py", line 739, in answer_challenge
    raise AuthenticationError('digest sent was rejected')
multiprocessing.context.AuthenticationError: digest sent was rejected
---------------------------------------------------------------------------

Is there any suggestion I can append EventProxy object in ListProxy?

Tobiichi
  • 31
  • 6

1 Answers1

0

Problem Solved! By following similar problem in Python 2.7 , we know that when python pickle proxied with multiprocessing after process initialization, there is no authkey available after unpickle. So we face the AuthenticationError.

One simple way to fix it:

import torch.multiprocessing as mp
mp.current_process().authkey = b'abc'

just add mp.current_process().authkey = b'abc' under your import.

Tobiichi
  • 31
  • 6