What's the best way to reassemble bytes sent from client to server?

Question

What's the best way to stream bytes from client to server in chunks of determined size?

Right now I'm encoding an audio file with base64, then compressing it with zlib and sending through the socket connection. My problem is trying to rebuild the original within the server.

I thought and tested using an empty string that is added with all the bytes the server is receiving. Seemed alright, but the " b' " in the beginning was being kept, which left it unable to recover the original audio file.

I've just tried to decode the bytes and deleting the " b' " from the beginning and " " " from the end (data[2:-1]) of each set of strings received by the server, but this cut a few characters from the original.

client side:

with open(arquivo, 'rb') as input_file:
   abc = base64.b64encode(input_file.read())

try1 = zlib.compress(abc)

n = 338
result = [try1[i:i+n] for i in range(0, len(try1), n)]

HOST = ''
PORT = 9999

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.connect((HOST,PORT))

i = 0

for item in result:
    item = str(item)
    print(item)
    s.send(item.encode())
    i += 1
    print('i = ', i)
    time.sleep(2)

Server side:

HOST = ''
PORT = 9999

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

print('Servidor Inicializado')
s.bind((HOST,PORT))
s.listen()
audiofile = ''
i = 0
conn, addr = s.accept()



while True:


    data1 = conn.recv(2048)
    print('data1 undecoded = ', data1)
    text = data1.decode()

    data = text[2:-1]
    print('data EDITADO = ', data)

    audiofile = audiofile + data
    i += 1
    print('i = ', i)
    print('audiofile = ', audiofile)

    if not data:
        print('No Data Received!')

    print('Recebeu tratado :', data)

No idea how to proceed, any help is appreciated. Thanks!

Somebody can correct me if I'm wrong about large files but I typically use `sendall` instead of sending in chunks. Prior to sending the file though, I will send a fixed length string indicating the full file size I'm sending. Then from the client side I receive the fixed file length, then receive the file in smaller chunks until I've received the entire thing. — Axe319, Jan 16 '20 at 17:52

Axe319 · Answer 1 · 2020-01-16T18:28:00.097

Here is an example of how I send and receive data with sockets.

Typically I'll pickle them. If you're not familiar with pickle it's used to serialize python objects to store or send over connections such as sockets.

Client Side:

import pickle

with open(arquivo, 'rb') as input_file:
   abc = base64.b64encode(input_file.read())
# I haven't used these libraries so I'm assuming you know how to unpack it from here
try1 = zlib.compress(abc)

HOST = ''
PORT = 9999

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.connect((HOST,PORT))

# serialize the python object 
message = pickle.dumps(try1)

# get the length of the pickled object
length = len(message)
# convert into a fixed width string
length = str(length).rjust(8, '0')
# send the length of the object we will send
s.sendall(bytes(length, 'utf-8'))

# send the object
self.client.sendall(message)

Server Side:

import pickle

HOST = ''
PORT = 9999

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

s.bind((HOST,PORT))
s.listen()

conn, addr = s.accept()

# get the length of the object we are about to receive
length = conn.recv(8)
# turn it back into an int
length = int(length.decode('utf-8'))

# I keep this to determine if we've received everything or not
full_length = length
message = None

# loop until we've zeroed out our length
while length > 0:
    # only receive what we need
    # at a maximum of 128 bit chunks
    chunk_len = min(128, length)

    length -= chunk_len

    chunk = conn.recv(chunk_len)
    if message is None:
        message = chunk
    else:
        message = message + chunk

# Edit: I've had issues with slow connections not receiving the full data
# for those cases adding something like this works
while len(message) < full_length:
    chunk_len = min(128, full_length - len(message))
    chunk = conn.recv(chunk_len)
    message = message + chunk

# now that we've received everything, we turn it back into a python object
try1 = pickle.loads(message)

# this should be the same try1 you sent

Disclaimer: I did not test any of this, nor do I know what the try1 object is or what you want to do with it. this is just getting it from point a to point b.

Neat! That's some approach that would make it easier than reassembling the bytes. I'm working with that, trying to see if the bytes format is not maintained on the server side. Thanks, Axe! Best Regards! — vaughan is god, Jan 16 '20 at 18:38

What's the best way to reassemble bytes sent from client to server?

1 Answers1