What is the use of StandardOpenOption.SPARSE?

Question

Java 7 defines this option, yet I fail to understand its usefulness.Consider this simple program, run on a recent enough Linux machine, with a Java 6 JVM:

public static void main(final String... args)
    throws IOException
{
    final long offset = 1L << 31;
    final RandomAccessFile f = new RandomAccessFile("/tmp/foo", "rw");
    f.seek(offset);
    f.writeInt(2);
    f.close();
}

When I query the file "shell wise", I get, as expected:

$ cd /tmp
$ stat --format %s foo
2147483652
$ du --block-size=1 foo
4096    foo

That is, the inode truthfully declares that the file has a size close to 2 GB, but its disk usage is in fact a single block since the underlying fs has a 4k block size. Good.

But I didn't need Java 7's StandardOpenOption.SPARSE for that. In fact, if I run this exact same code with a Java 7 JVM, the results do not vary.

Now, on to some Java 7-only code:

public static void main(final String... args)
    throws IOException
{
    final ByteBuffer buf = ByteBuffer.allocate(4).putInt(2);
    buf.rewind();

    final OpenOption[] options = {
        StandardOpenOption.WRITE,
        StandardOpenOption.CREATE_NEW
    };
    final Path path = Paths.get("/tmp/foo");
    Files.deleteIfExists(path);

    try (
        final SeekableByteChannel channel
            = Files.newByteChannel(path, options);
    ) {
        channel.position(1L << 31);
        channel.write(buf);
    }
}

This also creates a sparse file, and I did not have to specify StandardOpenOption.SPARSE at all.

So, what is it used for? Is there any OS/filesystem combination where this option actually influences the behaviour?

since you pointed out that you are working on a linux. so what file system are you using?ext4 or ext3? Because I want to know since create a file with java on linux will be sparse by default, but how can I create the same file with non-sparse in java? — Ryan Zhu, Nov 13 '13 at 05:11

score 6 · Accepted Answer · answered Jul 18 '13 at 07:41

6

Oracle's notes in the I/O tutorial list NTFS as one filesystem in which the option matters. Microsoft's docs on sparse file support in NTFS says that sparse files must be explicitly marked as sparse, and it lists actions specific to sparse files (zeroing out regions, searching for ranges with non-zero data, etc).

I don't have a Windows box handy on which to try this out, but seeing as the tutorials specifically call out NTFS, that might be a place to focus the search.

answered Jul 18 '13 at 07:41

yshavit

42,327
7
87
124

1

It's like said, the option makes a difference on NTFS: Without it one has lazy allocation, but writing at the end of a large file physically allocates 0-bytes for the whole file. With that option it doesn't, e.g. a 4 GiB file with 1 byte written at the end consumes only ~64 KiB of data on the disk. https://github.com/frohoff/jdk8u-jdk/blob/master/src/windows/classes/sun/nio/fs/WindowsChannelFactory.java#L331 – Thorsten Schöning Dec 21 '18 at 17:38

score 0 · Answer 2 · answered Jul 13 '13 at 21:32

0

When you use a RandomAccessFile like this on Linux it is Sparse by default on ext4, however if you try ext3 or Windows you may find it is not sparse. i.e. the behaviour of the plain RandomAccessFile constructor is OS dependent.

answered Jul 13 '13 at 21:32

Peter Lawrey

525,659
79
751
1,130

Uh, ext3? No sparse support? ;) Sorry but no, it has supported sparse since its inception, and so has ext2 for that matter (`open(2)` and `seek(2)` haven't changed). What is more, my fs is btrfs. As to Windows, well, are there options when you open/create a file to create it sparse or not? – fge Jul 13 '13 at 21:35
@fge If it didn't support sparse files, the option would be pointless, the option is to switch the default. – Peter Lawrey Jul 13 '13 at 21:38
Wait wait wait. Ext2+ have had support for sparse files since the beginning. You don't need to open a file with this option in order for the created file to be sparse (and as I said, `open(2)` has no such option -- neither does `fopen(2)` for that matter). My question really is, _when_ does this option make sense? In what context? As far as I know, it has no effect at all on any well behaved Unix system – fge Jul 13 '13 at 21:40
OK, let me rephrase: since POSIX has existed, neither `open(2)` nor `fopen(2)` have had a flag to open a file with an option saying, "this will be a sparse file". And each and every Unix filesystem having seen the day in the last 15+ years (ext*, btrfs, ufs, you name it) supports sparse files from the get go. Hence my question: why does Java have that flag? – fge Jul 13 '13 at 21:47
Some options in Java are added for "forward compatibility" for lack of a better word. It is possible it is one of these, options which end up doing nothing because they never did anything and then this get burned in for backward compatibility reasons. The only way to know for sure is to trace through the code, a simpler option would be `strace` and see if the system calls are any different on LUNIX at least. – Peter Lawrey Jul 14 '13 at 19:54

What is the use of StandardOpenOption.SPARSE?

2 Answers2

Linked