Python 2.7 multiprocessing - cant pass socket object via Manager dictionary

Question

I have a simple program that searches for open ports in a local network and stores the connected sockets in a dictionary along with their local address. Now, I am using a Manager shared dictionary to store these entries but it only accepts simple objects and not socket instances. Here is the code:

from multiprocessing import Process, Manager
import socket

manager = Manager()
# Store connected sockets
sockets = manager.dict()


def ping_addr(addr=None, port=None, timeout=None):
    """
    Create a socket and try to establish a connection to a specific address. If a connection is established, append
    the socket to the sockets dictionary.
    :param addr: The address.
    :param port: The port number.
    :param timeout: How many seconds to wait until its decided that the connection has been refused.
    :return: None
    """
    global sockets
    # Setup the client socket
    csocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    csocket.settimeout(timeout)
    # Try to connect
    try:
        csocket.connect((addr, port))
        print 'connected to {}:{}'.format(addr, port)
        # This works
        sockets.update({addr: 0})
        # This doesnt work
        sockets.update({addr: csocket})
    except socket.error:
        pass


for i in range(256):
    proc = Process(target=ping_addr, kwargs={'addr': '192.168.1.{}'.format(i), 'port': 14540, 'timeout': 0.5})
    proc.start()

The error I get is:

Process Process-4:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/alex/PycharmProjects/proj/test.py", line 29, in ping_addr
    sockets.update({addr: csocket})
  File "<string>", line 2, in update
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 758, in _callmethod
    conn.send((self._id, methodname, args, kwds))
TypeError: expected string or Unicode object, NoneType found

I did a short research about proxies but couldn't find a way to implement them into my code, so I am asking for help. How do I make socket instances compatible with a Manager dictionary?

georgexsh · Accepted Answer · 2017-10-30T07:31:37.657

3

passing socket from parent to child process is a very common practice. starting with python3.4, you could share a socket object directly via multiprocessing.Manager, but in python2.7, you have to wire it by yourself(see full gist here):

import socket
import copy_reg
from multiprocessing.reduction import rebuild_socket, reduce_socket

copy_reg.pickle(socket.socket, reduce_socket, rebuild_socket)

edited Oct 30 '17 at 07:31

answered Oct 30 '17 at 07:24

georgexsh

15,984
2
37
62

score 0 · Answer 2 · answered Oct 29 '17 at 18:21

0

Sockets cannot be passed to another process via Manager.dict. Each python socket has a socket at OS level. The OS knows binding between socket and process and it maintain socket ownership. The socket can be used only in process that created it. Or it may be used in child process at unix like system.

See also Passing Python object to another Python process

answered Oct 29 '17 at 18:21

Zaboj Campula

3,155
3
24
42

you state "sockets cannot be passed to another process", but you referenced a link offered an opposite opinion... – georgexsh Oct 30 '17 at 07:35

Python 2.7 multiprocessing - cant pass socket object via Manager dictionary

2 Answers2