5

I'm implementing a file transfer server, and I've run into an issue with sending a file larger than 2 GB over the network. The issue starts when I get the File I want to work with and try to read its contents into a byte[]. I have a for loop :

for(long i = 0; i < fileToSend.length(); i += PACKET_SIZE){
    fileBytes = getBytesFromFile(fileToSend, i);


where getBytesFromFile() reads a PACKET_SIZE amount of bytes from fileToSend which is then sent to the client in the for loop. getBytesFromFile() uses i as an offset; however, the offset variable in FileInputStream.read() has to be an int. I'm sure there is a better way to read this file into the array, I just haven't found it yet.

I would prefer to not use NIO yet, although I will switch to using that in the future. Indulge my madness :-)

Eliezer
  • 7,209
  • 12
  • 56
  • 103
  • What is your file transfer server written with? Do you use servlets? If so, just use the `HttpServletResponse.getOutputStream()` method and stream the file contents directly to the output stream...you don't have to do any chunking work youself. – Michael Nov 24 '11 at 14:31
  • I'm using `Sockets` and `ServerSockets`, which as far as I know, can only send `byte[]`. Am I missing something? – Eliezer Nov 24 '11 at 15:01
  • No, you're not using servlets :) I think I see your problem now. Because the files are so big, an `int` becomes too small. Let me write an answer – Michael Nov 24 '11 at 15:05
  • You are most likely to find the most efficient buffer size is around 32 KB (The size of your L1 cache) – Peter Lawrey Nov 24 '11 at 15:10

2 Answers2

4

It doesn't look like you're reading data from the file properly. When reading data from a stream in Java, it's standard practice to read data into a buffer. The size of the buffer can be your packet size.

File fileToSend = //...
InputStream in = new FileInputStream(fileToSend);
OutputStream out = //...
byte buffer[] = new byte[PACKET_SIZE];
int read;
while ((read = in.read(buffer)) != -1){
  out.write(buffer, 0, read);
}
in.close();
out.close();

Note that, the size of the buffer array remains constant. But-- if the buffer cannot be filled (like when it reaches the end of the file), the remaining elements of the array will contain data from the last packet, so you must ignore these elements (this is what the out.write() line in my code sample does)

Michael
  • 34,873
  • 17
  • 75
  • 109
  • Thanks a lot! This snippet is really gonna come in handy :) – Eliezer Nov 24 '11 at 15:21
  • 1
    @Eliezer And choosing your own 'packet size' is futile. The data will be buffered into the local socket send buffer, sent to the receiver in MTU-sized chunks, and then be buffered into the socket receive buffer. All traces of your 'packet size' disappear before the data even gets onto the wire. Just think of the buffer above as a file reading buffer. 32k is a good size as Peter Lawrey suggested. – user207421 Nov 24 '11 at 21:10
  • @EJP Someone suggested to me that 4096 was a good buffer size when reading/writing to disk because that's the block size that many file systems use. 32k sounds good too though. :) – Michael Nov 24 '11 at 21:14
  • @Michael That was true about 20 years ago. Disk subsystems have so much caching in them these days that cluster sizes are fairly irrelevant. It's more about getting enough data in one hit to keep the output buffer full. Many years ago I dealt with a printer spooler that printed one line at a time. It hogged the CPU because it got scheduled for every line, so under heavy load the system ran slowly, and the printer ran *very* slowly. We switched to a spooler that wrote 8192 byte chunks, which only got scheduled 132/8192 as often: the system ran faster and the printer ran *much* faster. – user207421 Nov 24 '11 at 21:39
  • @EJP So I would initialize the buffer straight to 32k or would I have to do something like `socket.setSendBufferSize(32k)` first? I use PACKET_SIZE so in case that value ever has to change for whatever reason, it wouldn't be a big deal. Something I carried over from school after having it drilled into me :) – Eliezer Nov 25 '11 at 01:13
  • @Eliezer You should do both. Socket buffer sizes on Windows are notoriously too small. If you're in control of the other end set a large socket receiver buffer too. I agree with your school about naming important constants; what I'm telling you is that the effect of this particular constant doesn't propagate beyond the write(). – user207421 Nov 25 '11 at 01:34
-1

Umm, realize that your handling of the variable i is not correct..

Iteration 0: i=0
Iteration 1: i=PACKET_SIZE
...
...
Iteration n: i=PACKET_SIZE*n
EdChum
  • 376,765
  • 198
  • 813
  • 562
user173438
  • 171
  • 1
  • 2
  • 8
  • I'm not exactly sure what you're saying. Care to explain? – Eliezer Nov 24 '11 at 15:02
  • It's exactly the value I would have wanted it to be if I didn't have the correct answer above. Why don't you tell me why that would be a problem instead of being cryptic? – Eliezer Nov 24 '11 at 15:43