Pyro4 Encoding in JSON/Base64

Question

So I have a simple application using Pyro4 (Python Remote Objects). There is an exposed class API, and I have a file that calls the function api.parse(input,userid), which returns some JSON dependent on input. However, instead of simply returning the result as a string (which it did previously), it returns {'data': 'eyJlcnJvciI6ICJDb21tYW5kIG5vdCByZWNvZ25pc2VkIn0=', 'encoding': 'base64'}, where the base64 is the JSON string that parse should returned.

I'm very confused as to why this is not working - I have tested it previously and there was no issue, the string was just returned with no weird base64 encoding. Only thing I can think of is I have changed networks (School connection to home connection) but I don't think this should be a problem? I have prepared an MVE of some code that indicates the problem.

testserver.py

import Pyro4;
import json;

@Pyro4.expose
class API:
    def parse(self,buff,userid):
        return prep({"error":"Command not recognised"});

def prep(obj):
    return json.dumps(obj).encode("utf-8");

# Pyro stuff
daemon = Pyro4.Daemon()                # make a Pyro daemon
ns = Pyro4.locateNS()                  # find the name server
uri = daemon.register(API)   # register the greeting maker as a Pyro object
ns.register("testAPI", uri)   # register the object with a name in the name server
daemon.requestLoop()

testclient.py

import Pyro4;
import json;

api = Pyro4.Proxy("PYRONAME:testAPI");
resp = api.parse("whatever","something");
print(resp); # Produces {'data': 'eyJlcnJvciI6ICJDb21tYW5kIG5vdCByZWNvZ25pc2VkIn0=', 'encoding': 'base64'}
# I just want b'{"error":"Command not recognised"}'

Note - Printing at the stage where prep is applied in parse() gives the expected result b'{"error":"Command not recognised"}'. I'm using the command python3 -m Pyro4.naming to start the nameserver if that matters as well. I'm thinking there's probably some global setting/constant I haven't set correctly or something - All responses welcome, thankyou!

Also just a note - yes I could just decode the base64 string but I want to know *why* it's doing this (And if I write code to decode base64 and it switches back I'll have to re-debug it). — James Paterson, Mar 01 '17 at 21:33

score 1 · Accepted Answer · answered Mar 02 '17 at 18:14

The default serialization protocol that Pyro uses is serpent, and that is a text-based protocol. This means it cannot transmit binary data (bytes) unless it encodes them to a text format, which it does in the way you discovered.

There's a little helper function (serpent.tobytes) available that you could use in your client code that automatically detects and converts the response if needed, see the info box in this paragraph in the manual: https://pythonhosted.org/Pyro4/tipstricks.html?highlight=base64#binary-data-transfer-file-transfer

Ofcourse, if you make the data sent by your server strings in the first place, there's no need for this. (This can't be done if the data really is binary though, such as an image or sound clip or whatever)

In your case with the json text it gets transformed into bytes if you encode it. As you discovered there is no need for this at all, just leave it as a string and you won't run into issues. (which leaves me with the question why you're still doing the encode to utf-8 bytes in your client?)

Thanks for the clearer and more detailed answer. The data is later sent over a socket connection (which needs bytes) , so I wanted to do all the processing in a single step. — James Paterson, Mar 02 '17 at 18:19
Ok, there's one other thing: why are you json-encoding at all in your server? Just return the response object as-is, and let the client decide what to do with it! — Irmen de Jong, Mar 02 '17 at 18:25
It's slightly more complicated. I have this server/client configuration, which then sends to another client. Eg: Datastore ---(Pyro)---> Frontend ---(Socket)---> Client, hence why I'm not just using the RMI object directly. — James Paterson, Mar 02 '17 at 19:15

score 0 · Answer 2 · answered Mar 01 '17 at 22:10

Turns out Pyro doesn't like trying to send raw bytes - It'd rather convert the bytes to base64 then send it as JSON. So to fix, I changed:

def prep(obj):
    return json.dumps(obj).encode("utf-8");

to

def prep(obj):
    return json.dumps(obj);

And put the encode bit in the client.

Pyro4 Encoding in JSON/Base64

testserver.py

testclient.py

2 Answers2