2

I've written a new IRC client to which I just added the DCC SEND part, therefore supporting direct file transfer for both users of the app. Nothing, fancy, I'm using the irc python library to power the client and Django for the GUI. The great miniupnpc lib takes care of port forwarding. However, whilst the file is properly being sent/received, the speed is absolutely HORRENDOUS : 20 KB/s approximatively. To test the server, I sent a package using Hexchat : the upload speed was the maximal theoretical bandwidth speed (in other words excellent). I tried looking for a buffer of some sort I may have missed. In the end, I must say I have absolutely no idea why my upload speed is so crappy and need some insight. Here is the relevant part of my upload script.

def on_dcc_connect(self, c, e):
    t = threading.Timer(0, upload_monitoring, [self, c])
    t.start()    
    log("connection made with %s" % self.nickname).write()
    self.file = open(self.filename, 'rb')
    self.sendBlock()


def sendBlock(self):
    if self.position > 0:
        self.file.seek(self.position)
    block = self.file.read(1024)
    if block:
        self.dcc.send_bytes(block)
        self.bytesSent = self.bytesSent + len(block)
    else:
        # Nothing more to send, transfer complete.
        self.connection.quit()

def on_dccsend(self, c, e):
    if e.arguments[1].split(" ", 1)[0] == "RESUME":
        self.position = int(e.arguments[1].split(" ")[3])
        c.ctcp("DCC", self.nickname, "ACCEPT %s %d %d" % (
        os.path.basename(self.filename),
        self.eport,
        self.position))



def on_dccmsg(self, connection, event):
    data = event.arguments[0]
    bytesAcknowledged = struct.unpack("!Q", data)[0]
    if bytesAcknowledged < self.bytesSent:
        return
    elif bytesAcknowledged > self.bytesSent:
        self.connection.quit()
        return
    self.sendBlock()

The send_bytes(block) method is the basic socket.send() method. When I increase the buffer of file.read(), I get struct.pack error, because the client's block reception acknowledgment (also struct.pack) is not properly read by my send script: data not of bytes length 8. Is it the file.read buffer that has to be changed? If so, why is the bytes received acknowledgment not the same at the sender's side as downloader's side? If not, where should I look to improve the upload speed?

mrj
  • 589
  • 1
  • 7
  • 17
  • 1
    We can't really say anything about this since we have no way to reproduce the performance you see. My guess is: reading a file in 1KB chunks and sending said chunks as 1KB packets will obviously result in very low performance. At the very least the file could be read in chunks of 64/128/512 KB and try to increase the size of the packets you send on the network as much as possible. – Bakuriu Jul 28 '16 at 19:42
  • In that case, would you have any idea as for why the bytesAcknowledged isn't the same as the acknoldgement sent by the client ? It's just `structured = struct.pack("!Q", self.dict[bot]["received_bytes"])` and then `connection.send_bytes(structured)` – mrj Jul 29 '16 at 14:30
  • I don't understand what you are asking me. Note that when sending data via sockets it can happen that *the kernel* decides to divide the message into more packets, this will result in the client reading less data. It's your responsibility to perform multiple calls until you have received all the data you need. – Bakuriu Jul 29 '16 at 14:36
  • What I mean is that when I increase the file buffer, I suddenly read a way bigger dccmsg than what I should. Does that mean that I have a socket buffer of some sort that first compiles data before flushing ? – mrj Jul 29 '16 at 14:40

1 Answers1

0

As I already suspected and as Bakuriu pointed out, the problem did indeed lie at the file.read(buffer) line. I finally found out why I had struct.pack error : the bytes acknowledgment was properly sent back to the sender, but sometimes a few packets were joined together. That is, for every packet received, an acknowledgment is answered to the sender in form of an 8 bytes length packed unsigned integer. Sometimes, sock.recv() doesn't read the incoming data fast enough and then, instead of having a bytes object of length 8, I have a bytes object of length 16, 24, 32, 40 or more. That's why I couldn't just unpack with struct.pack("!Q", data). Once I had that figured out, the solution was fairly easy to find :

def on_dccmsg(self, connection, event):
    data = event.arguments[0][-8:]
    bytesAcknowledged = struct.unpack("!Q", data)[0]

I just read the last 8 bytes from the data read by sock.recv() instead of reading everything. Now it works like a charm and the upload speed is the maximal theoretical upload speed allowed by my bandwidth !!!

mrj
  • 589
  • 1
  • 7
  • 17