Missing bytes while transferring file over SocketChannel

Question

I am transferring large file between Client and Server using nio ServerSocketChannel and SocketChannel.The problem is when I transfer file of size 6060064 bytes from the sender to the receiver, the receiver receives only 6059040 bytes missing some bytes.As the file size becomes bigger the difference gap of missing bytes increases.I can not find why are these bytes missing.

Senders Code :

public boolean send(SelectionKey selectionKey) {

    File file = new File("/media/data1/sample.mp4");
    try {
        SocketChannel socketChannel = (SocketChannel) selectionKey
                .channel();
        FileInputStream fileInputStream = new FileInputStream(file);
        System.out.println("File Length :: " + file.length());
        System.out.println("File lastModified :: " + file.lastModified());

        FileChannel fileChannel = fileInputStream.getChannel();

        transfer(fileChannel, socketChannel, file.length(), 1024 * 50);

        fileInputStream.close();

        System.out.println("File Send Completely Done 100%....");
        System.out.println("Closing Connection 100%....");

        return true;

    } catch (Exception e) {
        System.out.println(e);
        System.out.println("Connection Failed. Connectiontimeout.");
        return false;
    }
}

public static void transfer(FileChannel fileChannel,
        SocketChannel socketChannel, long lengthInBytes,
        long chunckSizeInBytes)
        throws IOException {

    long overallBytesTransfered = 0L;
    long time = -System.currentTimeMillis();
    
    while (overallBytesTransfered < lengthInBytes) {

        long bytesTransfered = 0L;

        bytesTransfered = fileChannel.transferTo(
                overallBytesTransfered,
                Math.min(chunckSizeInBytes, lengthInBytes
                        - overallBytesTransfered), socketChannel);

        System.out.println("bytesTransfered :: " + bytesTransfered);

        overallBytesTransfered += bytesTransfered;

        System.out.printf(
                "overall bytes transfered: %s progress %s%%\n",
                overallBytesTransfered, Math.round(overallBytesTransfered / ((double) lengthInBytes) * 100.0));
    }

    time += System.currentTimeMillis();

    System.out.printf("Transfered: %s bytes in: %s s -> %s kbytes/s\n",
                overallBytesTransfered, time / 1000,
                (overallBytesTransfered / 1024.0) / (time / 1000.0));
    System.out.println("------- File transfer completed ------");

}

Receivers Code :

public boolean recieve(SelectionKey key) {
    SocketChannel socketChannel = null;
    Socket socket = null;

    try {
        socketChannel = (SocketChannel) key.channel();
        socket = socketChannel.socket();
        System.out.println(socketChannel.getRemoteAddress());

        // Save file destination
        String FileKey = "/media/data1/Test/sample1.mp4";
        // File size
        long sizeInBytes = 6060064;
        
        long timeStap = 1402301850000l;

        File file = new File(FileKey);
        FileOutputStream fileOutputStream = new FileOutputStream(file);
        FileChannel fileChannel = fileOutputStream.getChannel();

        transfer(fileChannel, socketChannel, sizeInBytes, 1024 * 100);

        fileOutputStream.close();

        file.setLastModified(timeStap);
        socket.close();
        return true;

    } catch (Exception e) {
        System.out.println("Connection Failed. Connectiontimeout.");
        e.printStackTrace();
        return false;
    }
}

public static void transfer(FileChannel fileChannel,
        SocketChannel socketChannel, long lengthInBytes, long chunckSizeInBytes)
        throws IOException {

    long overallBytesTransfered = 0L;
    long time = -System.currentTimeMillis();
    while (overallBytesTransfered < lengthInBytes) {
        
        long bytesTransfered = 0L;

        bytesTransfered = fileChannel.transferFrom(
                socketChannel,
                overallBytesTransfered,
                Math.min(chunckSizeInBytes, lengthInBytes
                        - overallBytesTransfered));

        System.out.println("bytesTransfered :: " + bytesTransfered);

        overallBytesTransfered += bytesTransfered;

        System.out.printf(
                "overall bytes transfered: %s progress %s%%\n", overallBytesTransfered,
                Math.round(overallBytesTransfered / ((double) lengthInBytes) * 100.0));

    }
    
    time += System.currentTimeMillis();

        System.out.printf("Transfered: %s bytes in: %s s -> %s kbytes/s\n",
                overallBytesTransfered, time / 1000,
                (overallBytesTransfered / 1024.0) / (time / 1000.0));
        System.out.println("-------Dateiübertragung fertig------");

}

Output Of Sender:

bytesTransfered :: 18464

overall bytes transfered: 6060064 progress 100%

Transfered: 6060064 bytes in: 119 s -> 49.67708595651809 kbytes/s

Output Of Receiver:

bytesTransfered :: 0

overall bytes transfered: 6059040 progress 100%

The while loop of the receiver continues till I manually stop the process and bytesTransfered always returns zero on receiver side while the size of the file increases to 6059040 bytes.

Please can anybody tell why are these bytes missing.

Answer 1

We are seeing the same thing on JDK7u60 (which we didn't see previously)

We noted that if the buffer was greater than about 200KB the SocketChannel closed before the receiver had time to read the socket.

Our only suggestion so far is to throttle the writes with a pause

int bytes = socketChannel.write(currentWriteBuffer);
if (bytes > 200000) {
    try {
        Thread.sleep(100 * (bytes / 200000));
    }
    catch (InterruptedException e) {
      // ignore 
    }
}

I haven't yet looked into the actual limits or actual delay but this worked for us. Note that I found you question while researching a better answer.