Unpacking objects individually from a TCP stream

Question

I would like to read one by one the objects coming from a TCP stream, preferably using the MessagePack library.

On one side I have a client that, at each iteration of a loop:

computes a point location (tuple)
packs that location and sends it through a socket

On the other side, a server that:

receive the data when client is detected
unpack that data

For now I am storing the data in a buffer on reception, then proceed to unpacking when the stream is over. My problem is that I need to unpack the tuples one by one as they are sent. In other words I would like to read the data in real time without putting it in a buffer.

Provided it is possible, how could I achieve this using MessagePack ?

-- client side --

#Python3.7
import socket
import msgpack
import math

HOST = "127.0.0.1"
PORT = 9000

den = 40
rad = 100
theta = math.tau / den

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
    sock.connect((HOST, PORT)) #connect to server

    for step in range(den):
        x = math.cos(i*theta) * rad
        y = math.sin(i*theta) * rad
        data = msgpack.packb((x, y), use_bin_type = True)
        sock.sendall(data)

-- server side --

#Jython2.7 <-- Python 2.7 compatible only
from io import BytesIO
import msgpack
import socket

HOST = "127.0.0.1"
PORT = 9000

buf = BytesIO()

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
connection, address = s.accept()


while True:
    try:
        data = connection.recv(1024)
        buf.write(data)

    except:
        buf.seek(0)
        unpacker = msgpack.Unpacker(buf, use_list=False, raw=False)
        for unpacked in unpacker:
            print(unpacked)
        buf = BytesIO()

score 0 · Accepted Answer · answered May 29 '20 at 09:34

0

See "Stream Unpacking" section in the README: https://github.com/msgpack/msgpack-python#streaming-unpacking

You can do like this:

unpacker = msgpack.Unpacker(use_list=False, raw=False)

while True:
    data = connection.recv(1024)
    if not data:
        break
    unpacker.feed(data)
    for unpacked in unpacker:
        print(unpacked)

answered May 29 '20 at 09:34

methane

469
3
5
11

Thank you very much for the swift reply. Is 1024 mandatory in this case ? Or can it be lowered (32 or 16) ? – solub May 29 '20 at 11:03
1024 is just an example. You can choose number which is good for your application. – methane May 30 '20 at 12:48

Unpacking objects individually from a TCP stream

1 Answers1