twisted xmlrpc and numpy float 64 exception

Question

I'm using numpy to to some staff and then serve the results via a twisted/XMLRPC server. If the result is a numpy float 64, I get an exception cause probably twisted can't handle this type. Infact is I downgrade the result to float32 with x=float(x), everything is ok. This is not so good cause if I forget this workaroud somewhere, it's a pain. Have you any better solution?

server:

from twisted.web import xmlrpc, server
import numpy as np

class MioServer(xmlrpc.XMLRPC):
    """
    An example object to be published.
    """

    def xmlrpc_test_np(self):
        return np.sqrt(2)

if __name__ == '__main__':
    from twisted.internet import reactor
    r = MioServer()
    reactor.listenTCP(7080, server.Site(r))
    reactor.run()

client:

import xmlrpclib

if __name__=='__main__':
    x=xmlrpclib.ServerProxy('http://localhost:7080/')
    print x.test_np()

Exception:

Traceback (most recent call last):
  File "C:\Users\Stone\.eclipse\org.eclipse.platform_4.3.0_1709980481_win32_win32_x86\plugins\org.python.pydev_2.8.2.2013090511\pysrc\pydevd.py", line 1446, in <module>
    debugger.run(setup['file'], None, None)
  File "C:\Users\Stone\.eclipse\org.eclipse.platform_4.3.0_1709980481_win32_win32_x86\plugins\org.python.pydev_2.8.2.2013090511\pysrc\pydevd.py", line 1092, in run
    pydev_imports.execfile(file, globals, locals) #execute the script
  File "C:\Users\Stone\Documents\FastDose\src\Beagle\Prove e test\xmlrpc_client.py", line 28, in <module>
    print x.test_np()
  File "C:\Python27\lib\xmlrpclib.py", line 1224, in __call__
    return self.__send(self.__name, args)
  File "C:\Python27\lib\xmlrpclib.py", line 1578, in __request
    verbose=self.__verbose
  File "C:\Python27\lib\xmlrpclib.py", line 1264, in request
    return self.single_request(host, handler, request_body, verbose)
  File "C:\Python27\lib\xmlrpclib.py", line 1297, in single_request
    return self.parse_response(response)
  File "C:\Python27\lib\xmlrpclib.py", line 1473, in parse_response
    return u.close()
  File "C:\Python27\lib\xmlrpclib.py", line 793, in close
    raise Fault(**self._stack[0])
xmlrpclib.Fault: <Fault 8002: "Can't serialize output: cannot marshal <type 'numpy.float64'> objects">

Answer 1

This has nothing to do with twisted. If you read the error message you posted you see near the end that the error arises in xmlrpclib.py.

The xml-rpc implementation uses marshal to serialize objects. However, the marshalling done by xml-rpc does not support handling third party objects like numpy.ndarray. The reason that it works when you convert to float is that the built-in float type is supported.

Before offering my solution, I should point out that this exact same thing has been asked elsewhere in several places easily found via google (1 2), and I am stealing my answers from there.

To do what you want, you can convert your numpy array to something that can be serialized. The simplest way to do this is to write flatten/unflatten functions. You would then call then call the flattener when sending, and the unflattener when receiving. Here's an example (taken from this post):

from cStringIO import StringIO
from numpy.lib import format

def unflatten(s):
    f = StringIO(s)
    arr = format.read_array(f)
    return arr

def flatten(arr):
    f = StringIO()
    format.write_array(f, arr)
    s = f.getvalue()
    return s        

An even simpler thing to do would be to call

<the array you want to send>.tolist()

on the sending side to convert to a python list, and then call

np.array(<the list you received>)

on the receiving side.

The only drawback of doing this is that you have to explicitly call the flattener and unflattener when you send and receive data. Although this is a bit more typing, it's not much, and if you forget the program will fail loudly (with the same error you already experienced) rather than silently doing something wrong.

I gather from your question that you don't like this, and would rather find a way to make numpy arrays work directly with xml-rpc without any explicit flattening/unflattening. I think this may not be possible, because the xml-rpc documentation specifically says that the only third party objects that can be serialized are new style classes with a __dict__ attribute, and in this case the keys must be strings and the values must be other conformable types.

So you see, if you want to support numpy arrays directly, it seems that you have to modify the way xml-rpc's marshalling works. It would be nice if you could just add some kind of method to a subclass of ndarray to support being marshalled, but it looks like that's not how it works.

I hope this helps you understand what's going on, and why the existing solutions use explicit flattening/unflattening.

-Daniel

P.S. This has made me curious about how to extend xml-rpc in python to support other types of objects, so I've posted my own question about that.

Answer 2

I have found this way: before sending the result, I pass it to funcion to_float()

def iterable(x):
    try:
        iter(x)
    except: # not iterable
        return False
    else: # iterable
        return True


def to_float(x):
    from numpy import float64,ndarray
    if type(x) == dict:
        res = dict()
        for name in iter(x):
            res[name] = to_float(x[name])
        return res
    elif type(x) == ndarray:
        return map(float, x.tolist())
    elif type(x) == float64:  
        return float(x)  
    elif iterable(x) and not isinstance(x,str):    
        res=[]
        for item in x:
            if type(item) == float64: 
                res.append(float(item))
            elif type(x) == ndarray:
                res.append(map(float, x.tolist()))
            else:
                res.append(item)
        return res
    else:
        return x

Answer 3

Tried for quite some time to transfer a 640x480x3 image with xmlrpc. Neither the proposed "tolist()" nor the "flatten" solution worked for me. Finally found this solution:

Sender:

def getCamPic():
  cam = cv2.VideoCapture(0)
  img = cam.read()
  transferImg = img.tostring()
  return transferImg

Receiver:

transferImg = proxy.getCamPic()
rawImg = np.fromstring(transferImg.data,dtype=np.uint8) 
img = np.reshape(rawImg,(640,480,3))

As I know my cam resolution the hard coding is not an issue for me. An improved version might have this information included in the transferImg?