Throughput of arrays of different sizes vary

Question

I have arrays like

byte[] b = new byte[10];
byte[] b1 = new byte[1024*1024];

I populate them with some values. Say,

for(i=0;i<10;i++){
     b[i]=1;
}
for(i=0;i<1024*1024;i++){
     b1[i]=1;
}

Then I write it to a RandomAccessFile and read again from that file into the same array using,

randomAccessFile.write(arrayName);
              and
randomAccessFile.read(arrayName);

When I try to calculate the throughput of both these arrays(using the time calculated for file read and write) of varying sizes(10 bytes and 1Mb), throughput appears to be more for 1MB array.

Sample Output:
Throughput of 10kb array: 0.1 Mb/sec.
Throughput of 1Mb array: 1000.0 Mb/sec.

Why does this happen? I have Intel i7 with quad core processor. Will my hardware configuration be responsible for this? If not what could be the possible reason?

I'm waiting for a real expert to answer this question but the reason is that I/O is usually performed at chunks. Your 10kb array fits for example in one 512 kb chunk which means it would take as much time to write as a 512kb array, this means that the speed per unit of storage is lower. — Benjamin Gruenbaum, Jan 31 '13 at 20:49
I calculate throughput as chunk/time taken; where chunk=(1*10)/1024; for 10 KB. For 1 MB, chunk=1; — Prasanna, Feb 01 '13 at 19:21

score 3 · Accepted Answer · answered Jan 31 '13 at 21:04

The reason for the big difference is the overheads involved in I/O that occur no matter what the size of data being transferred - it's like the flag fall of a taxi ride. Overheads, which are not restricted to java and includes many O/S operations, include:

Finding the file on disk
Checking O/S permissions on the file
Opening the file for I/O
Closing the file
Updating file info in the file system
Many other tasks

Also, disk I/O is performed in pages (size depends on O/S, but usually 2K), so I/O of 1 byte probably costs the same as I/O of 2048 bytes: A slightly fairer comparison would be a 2048 byte array with a 1Mb array.

If you are using buffered I/O, that can further speed up larger I/O tasks.

Finally, what you report as "10Kb" is in fact just 10 bytes, so your calculation is possibly incorrect.

Throughput of arrays of different sizes vary

1 Answers1