Possible encoding issues when serializing user defined type using MessagePack in Delphi?

Question

I'm trying to serialize a Record in Delphi by using MessagePack and then using ZeroMQ TCP protocol I send it to a Python server.

b'\xa8DataType\x01\xa4data\xbf{"major":1,"minor":0,"build":2}\x00'

I'm having trouble deserializing it on the server side. Any ideas why this is happening? Is it some kind of encoding issues? Thanks!

Update #1:

I use the messagepack library "QMsgPack" recommended on www.msgpack.org Here's some Delphi code. My user defined Records and an enum:

Version = Record
    major : Integer;
    minor : Integer;
    build : Integer;
end;

TDataType = (dtUnset, dtVersion, dtEntityDescription, dtEntityDescriptionVector, dtEntityState, dtEntityStateVector, dtCommandContinue, dtCommandComplete);

TPacket = Record
    DataType : TDataType;
    data : string;
end;

And the code to serialize the object:

begin
    dmVersion.major := 1;
    dmVersion.minor := 1;
    dmVersion.build := 1;

    lvMsg := TQMsgPack.Create;
    lvMsg.FromRecord(dmVersion);

    lvMsgString := lvMsg.ToString();

    packet.DataType := dtVersion;
    packet.data := lvMsgString;
    lvMsg.Clear;
    lvMsg.FromRecord(packet);
    lvbytes:=lvMsg.Encode;
    ZeroMQ.zSendByteArray(skt, lvbytes);

I then try to deserialize the received byte array in the python server which looks like this:

b'\xa8DataType\x01\xa4data\xbf{"major":1,"minor":0,"build":2}\x00'

by using umsgpack.unpack() and then print out the result in the result like this:

packet_packed = command.recv()

# Unpack the packet
umsgpack.compatibility = True
packet = umsgpack.unpackb( packet_packed )
print (packet) 
for item in packet:
    print (item)

and this is what I get printed out on the screen:

b'DataType'
68
97
116
97
84
121
112
101

I hope this helps! Thanks!

Update #2

Here is some server code on the python side. The VDS_PACKET_VERSION is a constant int set to 1.

    # Make sure its a version packet
    if VDS_PACKET_VERSION == packet[0]:

        # Unpack the data portion of the packet
        version = umsgpack.unpackb( packet[1] )

        roster = []
        if ( VDS_VERSION_MAJOR == version[0] ) and ( VDS_VERSION_MINOR == version[1] ) and ( VDS_VERSION_BUILD == version[2] ):
            dostuff()

With the current serialized object

b'\x82\xa8DataType\x01\xa4data\xbf{"major":1,"minor":1,"build":1}'

I get

KeyError: 0 on packet[0]

Why is that?

Answer 1

The packed data appears to be invalid.

>>> packet = { "DataType": 1, "data": "{\"major\":1,\"minor\":0,\"build\":2}"}
>>> umsgpack.packb(packet)
b'\x82\xa4data\xbf{"major":1,"minor":0,"build":2}\xa8DataType\x01'

The first byte is \x82 which, as can be seen in the specification, is a two entry fixmap.

Your packed data is missing that information, and launches straight in to a fixstr. So, yes, there could be a mismatch between your Delphi based packer and the Python based unpacker. However, when I take your Delphi code, using the latest qmsgpack from the repo, it produces the following bytes:

82A8446174615479706501A464617461
BF7B226D616A6F72223A312C226D696E
6F72223A312C226275696C64223A317D

Let's convert that into a Python bytes object. It looks like this:

b'\x82\xa8DataType\x01\xa4data\xbf{"major":1,"minor":1,"build":1}'

Now, that's quite different from what you report. And umsgpack can unpack it just fine. Note that the first byte is \x82, a two entry fixmap, just as expected. Yes, the entries are in a different order, but that's just fine. Order is not significant for a map.

So, I've been able to encode using qmsgpack in Delphi, and decode using umsgpack in Python. Which then suggests that this issue is really in the transmission. It looks to me as though there has been an off-by-one error. Instead of transmitting bytes 0 to N-1, bytes 1 to N have been transmitted. Note the spurious trailing zero in your received data.

---

In the comments you obverse that the data field is being coded as JSON and passed as a string. But you'd rather have that data encoded using MessagePack. So here's what to do:

1. In the Delphi code change the data field's type from string to TBytes. That's because we are going to put a byte array in there.
2. Populate the data field using Encode, like this: packet.data := lvMsg.Encode.
3. On the Python side, when you unpack data you'll find that it is an array of integers. Convert that to bytes and then unpack: umsgpack.unpackb(bytes(data)).