QThreads and xmlrpc client

Question

I'm trying to use xmlrpc client from many QThreads. To make sure that only one thread is using xmlrpc client I created lock with QMutex. My code:

import sys
import xmlrpclib
import threading
import time

from SimpleXMLRPCServer import SimpleXMLRPCServer

from PyQt4 import QtCore, QtGui

class MM(object):
    def __init__(self):
        self.lock = QtCore.QMutex()
        self.xmlrpc_client = xmlrpclib.ServerProxy('http://localhost:9092')

    def __getattr__(self, name):
        self.lock.lock()
        sys.stderr.write('locked, for %s\n' % name)
        print threading.current_thread()
        result = self.xmlrpc_client.__getattr__(name)
        sys.stderr.write('unlocked by %s\n' % name)
        self.lock.unlock()
        return result

class Server(QtCore.QThread):
    def __init__(self):
        QtCore.QThread.__init__(self)
        self.server = None

    def run(self):
        self.server = SimpleXMLRPCServer(("localhost", 9092), logRequests = False)
        def one():
            return 1
        self.server.register_function(one, 'one')
        self.server.serve_forever()
        print "SERVER DONE"


class Ask(QtCore.QThread):
    def __init__(self, mmInst):
        QtCore.QThread.__init__(self)
        self.mm = mmInst
        self._stopping = False

    def run(self):
        while not self._stopping:
            time.sleep(0.5)
            print self.mm.one()

    def stop(self):
        self._stopping = True
        self.wait()


def start_gui():
    app = QtGui.QApplication(sys.argv)

    server = Server()
    server.start()

    time.sleep(1)

    mm = MM()
    print mm.one()

    a1 = Ask(mm)
    a1.start()

    a2 = Ask(mm)
    a2.start()

    try:
        app.exec_()
    except KeyboardInterrupt:
        server.server.shutdown()

if __name__ == "__main__":
    start_gui()

But it doesn't work and I'm not sure why. Here is output:

locked, for one
<_MainThread(MainThread, started 1648)>
unlocked by one
1
locked, for one
<_DummyThread(Dummy-1, started daemon 356)>
unlocked by one
locked, for one
<_DummyThread(Dummy-2, started daemon 1480)>
unlocked by one
Traceback (most recent call last):
  File "H:\poker\repos\TestSuite\test.py", line 46, in run
    print self.mm.one()
  File "C:\Python27\lib\xmlrpclib.py", line 1224, in __call__
    return self.__send(self.__name, args)
  File "C:\Python27\lib\xmlrpclib.py", line 1575, in __request
    verbose=self.__verbose
  File "C:\Python27\lib\xmlrpclib.py", line 1264, in request
    return self.single_request(host, handler, request_body, verbose)
  File "C:\Python27\lib\xmlrpclib.py", line 1289, in single_request
    self.send_request(h, handler, request_body)
  File "C:\Python27\lib\xmlrpclib.py", line 1391, in send_request
    connection.putrequest("POST", handler, skip_accept_encoding=True)
  File "C:\Python27\lib\httplib.py", line 853, in putrequest
    raise CannotSendRequest()
httplib.CannotSendRequest
Traceback (most recent call last):
  File "H:\poker\repos\TestSuite\test.py", line 46, in run
    print self.mm.one()
  File "C:\Python27\lib\xmlrpclib.py", line 1224, in __call__
    return self.__send(self.__name, args)
  File "C:\Python27\lib\xmlrpclib.py", line 1575, in __request
    verbose=self.__verbose
  File "C:\Python27\lib\xmlrpclib.py", line 1264, in request
    return self.single_request(host, handler, request_body, verbose)
  File "C:\Python27\lib\xmlrpclib.py", line 1294, in single_request
    response = h.getresponse(buffering=True)
  File "C:\Python27\lib\httplib.py", line 1015, in getresponse
    raise ResponseNotReady()
httplib.ResponseNotReady

When using only one Thread works fine:

$ diff -u test.py.back test.py
--- test.py.back        2012-03-14 01:34:37.666425000 +0100
+++ test.py     2012-03-14 01:33:01.423265000 +0100
@@ -63,8 +63,8 @@
     a1 = Ask(mm)
     a1.start()

-    a2 = Ask(mm)
-    a2.start()
+    #a2 = Ask(mm)
+    #a2.start()

     try:
         app.exec_()
$ python test.py
locked, for one
<_MainThread(MainThread, started -1219930432)>
unlocked by one
1
locked, for one
<_DummyThread(Dummy-1, started daemon -1287918736)>
unlocked by one
1
locked, for one
<_DummyThread(Dummy-1, started daemon -1287918736)>
unlocked by one
1
locked, for one
<_DummyThread(Dummy-1, started daemon -1287918736)>
unlocked by one
1

When using two different instances of MM also works fine:

$ diff -u test.py.back test.py
--- test.py.back        2012-03-14 01:34:37.666425000 +0100
+++ test.py     2012-03-14 01:38:47.352862000 +0100
@@ -57,13 +57,13 @@

     time.sleep(1)

-    mm = MM()
-    print mm.one()
+    #mm = MM()
+    #print mm.one()

-    a1 = Ask(mm)
+    a1 = Ask(MM())
     a1.start()

-    a2 = Ask(mm)
+    a2 = Ask(MM())
     a2.start()

     try:
adam@sabayon /media/Nowy/poker/repos/TestSuite $ python test.py
locked, for one
locked, for one
<_DummyThread(Dummy-1, started daemon -1288275088)><_DummyThread(Dummy-2, started daemon -1298138256)>
unlocked by one

unlocked by one
11

locked, for one
<_DummyThread(Dummy-2, started daemon -1298138256)>
unlocked by one
locked, for one
<_DummyThread(Dummy-1, started daemon -1288275088)>
unlocked by one
1
1
locked, for one
<_DummyThread(Dummy-1, started daemon -1288275088)>
unlocked by one
locked, for one
1<_DummyThread(Dummy-2, started daemon -1298138256)>

unlocked by one
1

Is this your complete example of the code? I spot some various errors and I just wanted to make sure this wasn't a real issue — jdi, Mar 13 '12 at 23:34
Yes, It should be possible just copy & paste this code to get presented output. — Adam, Mar 13 '12 at 23:36

jdi · Accepted Answer · 2012-03-14T02:07:29.753

There were a couple problems that I saw with the code, but mainly the one that I was getting right away was that you were starting the client before the server was ready. When I instantiated the server first, then the client, I stopped getting connection errors.

I also removed your spinning while loop and instead just started the pyqt event loop. Also you werent really passing your MM instance to your Ask class (not that it mattered much for this example since you were using the same instance).

Anyways, here is my version that seems to function:

import sys
import xmlrpclib
import threading
import time

from SimpleXMLRPCServer import SimpleXMLRPCServer

from PyQt4 import QtCore, QtGui

class MM(object):
    def __init__(self):
        self.lock = QtCore.QMutex()
        self.xmlrpc_client = xmlrpclib.ServerProxy('http://localhost:9092')

    def __getattr__(self, name):
        self.lock.lock()
        sys.stderr.write('locked, for %s\n' % name)
        print threading.current_thread()
        result = self.xmlrpc_client.__getattr__(name)
        sys.stderr.write('unlocked by %s\n' % name)
        self.lock.unlock()
        return result

class Server(QtCore.QThread):
    def __init__(self):
        QtCore.QThread.__init__(self)
        self.server = None

    def run(self):
        self.server = SimpleXMLRPCServer(("localhost", 9092), logRequests = False)
        def one():
            return 1
        self.server.register_function(one, 'one')
        self.server.serve_forever()
        print "SERVER DONE"


class Ask(QtCore.QThread):
    def __init__(self, mmInst):
        QtCore.QThread.__init__(self)
        self.mm = mmInst
        self._stopping = False

    def run(self):
        while not self._stopping:
            time.sleep(0.5)
            print self.mm.one()

    def stop(self):
        self._stopping = True
        self.wait()


def start_gui():
    app = QtGui.QApplication(sys.argv)

    server = Server()
    server.start()

    time.sleep(.25)

    mm = MM()
    print mm.one()

    a1 = Ask(mm)
    a1.start()

    a2 = Ask(mm)
    a2.start()

    try:
        app.exec_()
    except KeyboardInterrupt:
        server.server.shutdown()


if __name__ == "__main__":
    start_gui()

Update

After looking into this a bit more, I realized that its a bug in python 2.7 and xmlrpc, where they changed how it creates connections. http://bugs.python.org/issue6907

Strangely, this code does not crash for me on python 2.6/2.7 on OSX, or on python 2.6 under linux. But it DOES crash for me with python 2.7 under linux.

When I moved the locking mechanism outside of the MM instance, it seemed to start working under 2.7 on linux:

class MM(object):
    def __init__(self):
        self.xmlrpc_client = xmlrpclib.ServerProxy('http://localhost:9093')

    def __getattr__(self, name):
        return self.xmlrpc_client.__getattr__(name)


class Ask(QtCore.QThread):
    def __init__(self, mmInst, lock):
        QtCore.QThread.__init__(self)
        self.mm = mmInst
        self.lock = lock

    def run(self):
        while not self._stopping:
            time.sleep(0.5)
            self.lock.lock()
            print self.mm.one()
            self.lock.unlock()

def start_gui():
    app = QtGui.QApplication(sys.argv)

    ...

    lock = QtCore.QMutex()

    a1 = Ask(mm, lock)
    a1.start()

    a2 = Ask(mm, lock)
    a2.start()

It looks like I have the same issue with your code (even if I increase sleep after starting server to 3 seconds)... http://pastebin.com/eBzaTbu4 — Adam, Mar 14 '12 at 00:06
And at beginning of output you can see printed 1, it means that server is ready because I'm doing print mm.one() after starting it. — Adam, Mar 14 '12 at 00:12
Strange. Works fine for me. So I cant reproduce it. There must be something specific with XMLRPC and the communication that needs to be caught and handled. — jdi, Mar 14 '12 at 00:23
Can you try and report the outcome of these things: 1. Try it with only one thread starting instead of two. 2. Try it with 2 unique instances of MM, one for each thread. — jdi, Mar 14 '12 at 00:32
Updated my post. Maybe it will be possible to reproduce it witch decreasing sleep time in run method of Ask? — Adam, Mar 14 '12 at 00:47
Actually turned off the sleep completely and let it run as fast as it could and it still wouldn't crash for me. :-/ — jdi, Mar 14 '12 at 00:53
I'm testing it with python 2.7.2, pyqt 4.8.5. Maybe here we have the difference? — Adam, Mar 14 '12 at 00:55
My guess is that it has to do with where the context switch happens in the shared memory? I think the best solution overall is to create your lock outside of the MM class completely, and then make your Ask class require a lock instance as parameter to the `__init__()` — jdi, Mar 14 '12 at 02:04
Updated one last time to show the example I was just talking about — jdi, Mar 14 '12 at 02:07

QThreads and xmlrpc client

1 Answers1

Update

Linked