0

I have a feeling that this is related to the absence of fork() in Windows, but I'm not really sure what to do about it...

I have a class called workObj, which inherits Process from multiprocessing. On Windows, creating a new instance of this class goes well, but calling the instance's start() method throws the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Python27\lib\multiprocessing\forking.py", line 381, in main
    self = load(from_parent)
  File "C:\Python27\lib\pickle.py", line 1384, in load
    return Unpickler(file).load()
  File "C:\Python27\lib\pickle.py", line 864, in load
    dispatch[key](self)
  File "C:\Python27\lib\pickle.py", line 1089, in load_newobj
    obj = cls.__new__(cls, *args)
TypeError: __new__() takes at least 2 arguments (1 given)

Why? And what steps must I take to fix it? The data its trying to pickle should just be strings, which are pickleable, right? Even so, if I was trying to pickle an unpicklable data type, shouldn't that fail on Linux, too?

workObj consists of an infinite loop that reaches out via python sockets to a server, retrieves a string, processes it, and sends it back to the server.

class workObj(Process):
        def __init__(self, name, ip, port, keyFile, *args, **kwargs):
                Process.__init__(self)
                global printer
                self.tprint = printer.tprint            # We'll hold a reference to printer internally.
                self.workCounter = 0
                self.ip = ip
                self.port = port
                self.conn = socket.socket()
                self.workDone = False
                self.connected = True
                self.conn.settimeout(10)
                self.secconn = ssl.wrap_socket(self.conn, ca_certs=keyFile, cert_reqs=ssl.CERT_REQUIRED)
                self.name = name
                try:
                        self.secconn.connect((self.ip, self.port))
                except:
                        printer.tprint(self.name, "Fatal error in constructor: workObj {}: connect to {}:{} failed! WARNING! Object is unusable!".format(self.name, self.ip, self.port), error= True)
                        self.workDone = True
                        self.connected = False
                        # raise                         # Come back later and check this syntax.
        def getWorkCount(self):
                return self.workCounter
        def isConnected(self):
                return self.connected
        def finishUp(self):
                self.workDone = True
        def run(self):
                s = self.secconn
                tprint = self.tprint
                try:
                        while not self.workDone:
                                workStr = s.recv(128)
                                if workStr == "":
                                        self.workDone = True
                                        tprint(self.name, "Got blank job from server.")
                                        break
                                sendStr = doWork(workStr)
                                s.send(sendStr)
                                #s.send(doWork(s.recv(128)))
                                self.workCounter += 1
                except socket.timeout as e:
                        tprint(self.name, "Timeout getting or sending work. Assuming server broke.", error= True)
                        tprint(self.name, str(e), error=True)
                        self.workDone = True
                except KeyboardInterrupt as e:
                        tprint(self.name, "Caught keyboard interrupt, sending server blank job.", error=True)
                        s.send("")
                        self.workDone = True
                except:
                        # Literally anything else?
                        import sys
                        e = sys.exc_info()[:]
                        tprint(self.name, "workObj {} broke.".format(self.name), error=True)
                        tprint(self.name, "{}\n{}".format(e[1], e[2]), error=True)
                        self.workDone = True
                        #raise
                s.close()
                self.connected = False
ADS103
  • 85
  • 1
  • 8
  • 2
    The `workObj` object you create in the main process needs to be copied into the child processes. On Linux that can be done using `fork`, makes the child process a complete copy of the parent process. There's no `fork` system call on Windows, so the `multiprocess` module has to use pickle to copy the `workObj` object into the child process. – Ross Ridge Jul 01 '17 at 16:09
  • See the `multiprocessing` module documentation for more details: https://docs.python.org/2/library/multiprocessing.html#windows – Ross Ridge Jul 01 '17 at 16:19
  • You've answered my question, thank you. I understand exactly what the problem is now, and indeed I've gotten around it using the jsonpickle library in the past - just not in a multiprocessing situation! When Windows creates the new process, it doesn't have a copy of the work object, and because the work object is a custom class, it can't be pickled. I suppose I can get around this by just making it a function... – ADS103 Jul 01 '17 at 16:42
  • Custom classes can be pickled. The problem is likely that one of more of the instance members you set in `workObj.__init__` can't be pickled. You might try moving the socket creation code, along with the initialization of `tprint`, to the `run` method. – Ross Ridge Jul 01 '17 at 16:58

0 Answers0