Making file transfer more efficient Java

Question

I have two wireless computers connected to an N wireless router. Each of these PCs are connected at between 108-150Mbps.

Theoretically, I should be able to transfer at 13.5MB/s to 18.75MB/s, under the absolute best of conditions.

The first computer (that is sending), uses a very fast SSD, which is around 100MB/s if I remember correctly. CPU usage also stays below 20%.

It sent 1960273535 bytes (1.8GB) in 656367ms. That's 2.8MB/s (22 out of 108 Megabits). When I open up task manager, I see that only 25-27% of the network connection is being used.

I am looking for any ideas, suggestions, or improvements that can make the transfer faster (over a network). I was thinking of buffering the file from the disk on a thread and sending the buffered data from another thread but I'm not sure if it's a good idea. Here is the SSCCE:

Host:

import java.io.*;
import java.net.*;


public class Host {


    public static void main(String[] args) throws IOException {
        
        
        ServerSocket servsock = new ServerSocket(15064);
        Socket sock = servsock.accept();
            long time = System.currentTimeMillis();

        OutputStream out = sock.getOutputStream();
        FileInputStream fileInputStream = new FileInputStream("C:\\complete.rar");

        byte [] buffer = new byte[64*1024]; 
        int bytesRead = 0;
        long totalSent = 0;
        
        while ( (bytesRead = fileInputStream.read(buffer)) != -1)
        {
            if (bytesRead > 0)
            {   
                out.write(buffer, 0, bytesRead);
                totalSent += bytesRead;
                System.out.println("sent " + totalSent);
            }   
        }
        
        sock.close();
        
        System.out.println("Sent " + totalSent + " bytes in "
                + (System.currentTimeMillis() - time) + "ms.");
        
    }
}

Client:

import java.io.*;
import java.net.*;

public class Client {

    public static void main(String[] args) throws Exception {
        Socket sock = new Socket("127.0.0.1", 15064);
        InputStream in = sock.getInputStream();
        FileOutputStream fileOutputStream = new FileOutputStream("output.rar");

        byte [] buffer = new byte[64*1024]; 
        int bytesRead = 0;

        while ( (bytesRead = in.read(buffer)) != -1)
            fileOutputStream.write(buffer, 0, bytesRead);
        sock.close();
        fileOutputStream.close();
    }
}

Edit: I tried mapping a network drive and sending the file over that, and windows did even worse - 2.35MB/s. According to this article mapping a network drive is faster than FTP, and I also don't have the time to stay playing around and setting up the FTP server.

Edit2: After changing the timer, turns out it was transferring at 3MB/s over WiFi. I hate the "theoretical" throughput. When I buy something, I want to know it's REAL performance. It turns out the code is indeed limited by WiFi speeds. I am still open to suggestions though.

Edit 3: After running the program on 100Mbps LAN, it managed to transfer the file at 11.8MB/s. That's pretty good considering that the maximum transfer rate is 12.5MB/s.

Answer 1

At 2.8MB/s, it is unlikely that the slowness has anything to do with your code. It is almost certainly due to the wireless network not being able to achieve the theoretical throughput (possibly due to environmental conditions).

It's easy to test whether this is the case: simply time a large ftp or scp file transfer between the same two computers and see what kind of throughput you're seeing.

Answer 2

I suggest you try the following code which prints

Wed Oct 26 14:21:03 BST 2011: Accepted a connection
Wed Oct 26 14:21:13 BST 2011: Transfer rate was 3212.5 MB/s

on the server and on the client prints

Wed Oct 26 14:21:03 BST 2011 Sending for 10.0 seconds.
Wed Oct 26 14:21:13 BST 2011 ... sent.
Wed Oct 26 14:21:13 BST 2011 ... received 33691287552
Send and received 3212.8 MB/s

Note: the total amount transferred is double this as everything sent client to server is sent server to client.

---

import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.ServerSocket;
import java.net.Socket;
import java.nio.ByteBuffer;
import java.nio.channels.ServerSocketChannel;
import java.nio.channels.SocketChannel;
import java.util.Date;

public class EchoServerMain {
  public static void main(String... args) throws IOException {
    int port = args.length < 1 ? 55555 : Integer.parseInt(args[0]);
    ServerSocketChannel ss = ServerSocketChannel.open();
    ss.socket().bind(new InetSocketAddress(port));
    while (!ss.socket().isClosed()) {
      SocketChannel s = ss.accept();
      System.out.println(new Date() + ": Accepted a connection");
      long start = System.nanoTime();
      ByteBuffer bytes = ByteBuffer.allocateDirect(32*1024);
      int len;
      long total = 0;
      // Thank you @EJP, for a more elegant single loop.
      while ((len = s.read(bytes)) >= 0 || bytes.position() > 0) { 
        bytes.flip(); 
        s.write(bytes); 
        bytes.compact(); 
        total += len;
      }
      long time = System.nanoTime() - start;
      System.out.printf(new Date() + ": Transfer rate was %.1f MB/s%n", total * 1e9 / 1024 / 1024 / time);
    }
    ss.close();
  }
}

and

import java.io.EOFException;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.Socket;
import java.net.UnknownHostException;
import java.nio.ByteBuffer;
import java.nio.channels.SocketChannel;
import java.util.Date;

public class EchoClientMain {
  public static void main(String ... args) throws IOException {
    String hostname = args.length < 1 ? "localhost" : args[0];
    int port = args.length < 2 ? 55555 : Integer.parseInt(args[1]);
    double seconds = args.length < 3 ? 10 : Double.parseDouble(args[2]);

    SocketChannel s = SocketChannel.open(new InetSocketAddress(hostname, port));
    s.configureBlocking(false);
    ByteBuffer bytes = ByteBuffer.allocateDirect(32*1024);

    System.out.printf(new Date()+ " Sending for %.1f seconds.%n", seconds);
    long start = System.nanoTime();
    long dataSent = 0, dataReceived = 0;
    // run for 10 seconds.
    while(start + seconds*1e9 > System.nanoTime()) {
      bytes.clear();
      int wlen = s.write(bytes);
      if (wlen < 0) throw new IOException();
      dataSent += wlen;

      bytes.clear();
      int rlen = s.read(bytes);
      if (rlen < 0) throw new EOFException();
      dataReceived += rlen;
    }
    System.out.println(new Date()+ " ... sent.");

    while(dataReceived < dataSent) {
      bytes.clear();
      int rlen = s.read(bytes);
      if (rlen < 0) throw new EOFException();
      dataReceived += rlen;
    }
    s.close();
    long time = System.nanoTime() - start;
    System.out.println(new Date()+ " ... received "+dataReceived);
    System.out.printf("Send and received %.1f MB/s%n", dataReceived * 1e9/1024/1024/time);
  }
}

Answer 3

Your timer is wrong ! you should start it after you accept the connection not when you start the host

Did you try increasing your buffer size ?

Answer 4

Test the same program using a wired, Fast Ethernet (100Mbit/s) link between the computers (and then possibly using a 1Gbit link). That way, you'll see whether the transfer rate is actually limited by your program or by the link.

Answer 5

Just set a very large socket send buffer, and if possible set a very large socket receive buffer at the receiver. Code 'optimizations' contribute basically nothing to these scenarios.

Set a very large socket receive buffer at the receiver, 256K or more. Use a BufferedOutputStream around the FileOutputStream so that you are always writing to block boundaries. NB your test if (bytesRead > 0) can never be false. The only way InputStream.read() can return zero is if you supply a zero length buffer or a zero length parameter, and you aren't. I would get rid of the println() calls inside the loops.