JAVA using RandomAccessFile after seek is very slow. What is the reason?

Question

This is my test code

long fileSize = 1024 * 1024 * 512L;
byte[] bts = new byte[8];

RandomAccessFile randomAccessFile = new RandomAccessFile("f:/test.data", "rw");
randomAccessFile.setLength(fileSize);

randomAccessFile.seek(0);
long time = System.nanoTime();
randomAccessFile.write(bts);
System.out.println("write1 use:" + (System.nanoTime() - time));

randomAccessFile.seek(1024 * 1024 * 256L);
time = System.nanoTime();
randomAccessFile.write(bts);
System.out.println("write2 use:" + (System.nanoTime() - time));

print

write1 use:181051
write2 use:2029338072

It can be seen that writing is 9 bytes twice and the second time is 10000 times slower than the first time.

So I would like to ask why the seek will cause the file to write so slowly. Is there any solution?

This has to create ~256MB of data when you do the seek the second time because of the huge offset. Why are you seeking so far ahead? — tadman, May 05 '18 at 15:28
I'm developing a high speed http downloader,need to download multiple chunks at the same time,so I want seek file on chunk down response coming. — liwei2633, May 05 '18 at 15:48
Test with smaller offsets to see if that delay is proportional. Alternatively, write out to chunked files first, then assemble after. — tadman, May 05 '18 at 16:03
What's your OS and filesystem type? For something like this to work and not be horribly slow, the OS and file system have to support [sparse files](https://en.wikipedia.org/wiki/Sparse_file). Otherwise, when you write to a large offset with no previous data in between, the system needs to create all the data as @tadman stated in his comment. Note also that you can use [FileChannel.write(ByteBuffer src, long position)](https://docs.oracle.com/javase/7/docs/api/java/nio/channels/FileChannel.html#write(java.nio.ByteBuffer,%20long)) to write to an arbitrary location in a file without seeking. — Andrew Henle, May 05 '18 at 19:42
Thx,my OS is windows 10 and filesystem type is NTFS,i try to use FileChannel.write(ByteBuffer src, long position),then still the same.But i use FileChannel.map(),this is very fast.However, there is a problem that the maped file cannot exceed 2GB. — liwei2633, May 06 '18 at 04:04
NTFS requires explicitly setting a flag to create a sparse file. See https://msdn.microsoft.com/en-us/library/windows/desktop/aa365566(v=vs.85).aspx See this question and the first answer for more details: https://stackoverflow.com/questions/17634362/what-is-the-use-of-standardopenoption-sparse — Andrew Henle, May 06 '18 at 11:49
Thank you very much.I try to use Files.newByteChannel() with StandardOpenOption.SPARSE option.It is very fast — liwei2633, May 07 '18 at 02:12

score 1 · Answer 1 · answered Sep 21 '20 at 15:18

What you want is to create a sparse file. https://en.wikipedia.org/wiki/Sparse_file

final ByteBuffer buf = ByteBuffer.allocate(4).putInt(2);
buf.rewind();

final OpenOption[] options = {
    StandardOpenOption.WRITE,
    StandardOpenOption.CREATE_NEW,
    StandardOpenOption.SPARSE
};
final Path path = Paths.get("/tmp/foo");
Files.deleteIfExists(path);

try (
    final SeekableByteChannel channel
        = Files.newByteChannel(path, options);
) {
    channel.position(1L << 31);
    channel.write(buf);
}

The code was taken from What is the use of StandardOpenOption.SPARSE?

JAVA using RandomAccessFile after seek is very slow. What is the reason?

1 Answers1