Python multiple file transfer over TCP sockets

Question

I am trying to write a program in python to transfer multiple files in a folder over a socket, I have the following code so far

Client:

def uploadFiles(folder,dcip,PORT,filetype):
    os.chdir(folder)
    dirList = os.listdir(folder)
    print dirList
    ms = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    print PORT
    ms.connect((dcip, int(PORT)))
    for fname in dirList:

        if fname.endswith(str(filetype)):
            cmd = 'get\n%s\n' % (fname)
            ms.sendall(cmd)
            f = open(fname,'rb')
            data = f.read()
            f.close()
            print data
            r = ms.recv(2)
            ms.sendall(data)
            ms.sendall('done\n%s\n' %(fname))
    ms.sendall('end\n\n')   
    ms.close()

Server:

import socket,os
listener_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
listener_socket.bind(('',33234))
filename = ''
while 1:
    listener_socket.listen(100)
    connection,address_client = listener_socket.accept()
    if not os.path.exists(str(address_client[0])):
        os.makedirs(str(address_client[0]))
    currdir = os.getcwd()
    os.chdir('./'+str(address_client[0]))

    while(1):
        data = connection.recv(4096)
        cmd = data[:data.find('\n')]

        if cmd == 'get':
            x,filename,x = data.split('\n',2)
            connection.sendall('ok')
            f = open(filename,'wb')


        if cmd == 'done':
            continue

        f.write(data)

        if cmd == 'end':
            connection.close()
            break
    os.chdir(currdir)

The above code goes into an infinite loop, which I do understand that it is because of the continue statement in the if cmd == 'done' part, I wonder why it is doing that? I mean it never gets the done message from the client, can anyone please help me fix the code?

Your loop will terminate on the 'end' message, not the 'done' message. — James Thiele, Feb 07 '13 at 21:00
I am sending an end message after transfering all the files. Also, it does not copy the files as it is, it sticks in a loop for the first file itself. — srnvs, Feb 07 '13 at 21:31
Since reading is done in blocks, only first command will be at beginning of block. Other commands (get, done, end) will be somewhere inside same or next block(s). More clever parsing of received data should be done. — Ante, Feb 08 '13 at 01:36

Ante · Accepted Answer · 2013-02-11T21:17:59.277

Because commands and filenames are in separate lines, it is best to parse received data by lines. Last part of block data doesn't have to finish with a \n, so it has to be merged with next received data block.

This is (not tested) implementation of parsing received data by lines:

data = '' # contains last line of a read block if it didn't finish with \n
in_get, in_done, reading_file, ended = False, False, False, False
while not ended:
  if len(data) > 100:  # < update
    f.write( data )    # <
    data = ''          # <
  data += connection.recv(4096)
  i = data.find('\n')
  while i >= 0 and not ended:
    line = data[:i]
    data = data[i+1:]
    if in_get:
      filename = line
      reading_file = True
      f = open(filename,'wb')
      in_get = False
    elif in_done:
      if line != filename:  # done inside file content
        f.write( 'done\n' + line + '\n' )
      else:
        f.close()
        reading_file = False
      in_done = False
    else:
      if line == 'get' and not reading_file:
        in_get = True
      elif line == 'done':
        in_done = True
      elif line == 'end' and not reading_file:
        ended = True
        break;
      else:
        f.write( line + '\n' )
    i = data.find('\n')

Ante, I made the required changes and it works like a charm for smaller files, but I am trying to transfer larger files (256KB text files, all random numbers with no new lines), it does not recognize the new lines between the files, it just writes everything into one large file and goes into an infinite loop, can you tell me where it could go wrong specially for large files? — srnvs, Feb 11 '13 at 21:05
I don't know :-) Commands and file names are 'short', so maybe fix with writing data to a file if data is too long can help. I will change upper code with that fix. — Ante, Feb 11 '13 at 21:16

Python multiple file transfer over TCP sockets

1 Answers1