How to recieve and parse data coming from a Python socket server in dart?

Question

I have a socket server running in Python that sends a byte stream through the socket in which, the first 8 bytes contain the size of the remaining message body. The code looks something like this:

    data_to_send = struct.pack("Q", len(pickle_dump))+pickle_dump
    
    client_socket.sendall(data_to_send)

The client endpoint of this, that is written in Python looks like this:

data = b''
PAYLOAD_SIZE = struct.calcsize('Q')

while len(data) < PAYLOAD_SIZE:
    packet = client_socket.recv(1024)
    if not packet:
        break
    data += packet

    print(len(data), PAYLOAD_SIZE)

packed_msg_size = struct.unpack('Q', data[:PAYLOAD_SIZE])[0]
data = data[PAYLOAD_SIZE:]

while len(data) < packed_msg_size:
    data += client_socket.recv(1024)

print(type(data))
frame_data = data[:packed_msg_size]
data = data[packed_msg_size:]

Now I am kind of new to dart and flutter. So I kind of tried recieving the packets through dart's sockets (which I am not sure how it is implemented) like this:

Socket.connect("10.0.2.2", 7878).then((socket) {
        SOCKET = socket;

        setState(() {
          print('Connected to: '
              '${socket.remoteAddress.address}:${socket.remotePort}');
          log = '${socket.remoteAddress.address}:${socket.remotePort}';
        });

        var FULL_CLUSTER = ByteData(0);


        // MAIN STUFF STARTS HERE
        socket.listen((data) {
          Uint8List size_bytesdata = data.sublist(0, 8);
          print(size_bytesdata.buffer.asByteData().getUint64(0)); // Prints -6552722064661282816
          print(size_bytesdata);
          socket.destroy();
        });
        //

      }).onError((error, stackTrace) {
        setState(() {
          log = '$error\n\n$stackTrace';
        });
      });

Now first of all, I figured out that Dart's sockets shows bytes in Uint8List format, so I tried doing what I did in the python client script, but the values do not match.

The len(pickle_dump) has the value 921765 but the dart code(as mentioned in the snippet) prints -6552722064661282816. Now I believe the type recognition by dart is what's causing the trouble as I have packed is as 'Q' (using struct) in the Server script, which stands for 'Unsigned long long' but, in dart, I parsed it as 'Unsigned int'. How do I solve this issue? Also, a brief discussion what's exactly going will help a lot.

If you are going to be reading arbitrary lumps of data from a stream, you will find this class particularly useful: https://pub.dev/documentation/buffer/latest/buffer/ByteDataReader-class.html Your `listen` just dumps the data on the end and you can read all sorts of data types from the front. — Richard Heap, Dec 30 '22 at 13:59
I mean, the funtion `getUint64(0)` is literally supposed to print an Unsigned int, but it's printing a negative value, i.e., *-6552722064661282816* What is going on? — Broteen Das, Dec 31 '22 at 07:00
Edit: I figured it out. The problem is that the endianness needs to be little in this case, i.e., `getUint64(0, Endian.little)`. I think it's got something to do with struct's packing, I'm not sure, I will post a detailed solution after I get a grasp of what is actually happening as a whole. — Broteen Das, Dec 31 '22 at 07:32
check https://docs.python.org/3/library/struct.html - they say: "Alternatively, the first character of the format string can be used to indicate the byte order, size and alignment of the packed data, according to the following table" - I would use `!` prefix to be sure that always the same order is used — pskink, Dec 31 '22 at 13:14

score 0 · Accepted Answer · answered Jan 02 '23 at 14:51

As mentioned by pskink, the problem here is caused because the default Endian of Dart's getUint64() is different than that of Python's struct library (which uses the 'little' endianness to pack data). So just rewritting the line fromgetUint64(0) to getUint64(0, Endian.little) fixes the problem in this case.

How to recieve and parse data coming from a Python socket server in dart?

1 Answers1