1

FileChannel#map, which is used to map a file (i.e., initiate a memory map), takes a long as length parameter.

Yet, the documentation on FileChannel#map says the following:

size - The size of the region to be mapped; must be non-negative and no greater than Integer.MAX_VALUE

First of all, why do they use a long in the first place if they only allow values up to Integer.MAX_VALUE? Is it somehow possible to map files larger than that at once?

For example, if I would like to map a 10GB file, I would like to write something like this (which ends up in an InvalidArgumentException due to the too large length):

long length = 10_000_000_000;
MappedByteBuffer buffer = (new RandomAccessFile("./memory.map", "rw")).getChannel().map(FileChannel.MapMode.READ_WRITE, 0, length);

Do I have to create consecutive memory maps for that, e.g., 5 memory maps each mapping 2GB of the given file?

Markus Weninger
  • 11,931
  • 7
  • 64
  • 137

2 Answers2

3

FileChannel#map will return an instance of ByteBuffer, (MappedByteBuffer) and if you look at it, you can see that it's methods exclusively use int to index the data, e.g:

abstract byte get(int index) Absolute get method.

Which defines the limit for map itself, too.

So, you would have to either map your file piece by piece or use some existing library, e.g. https://github.com/xerial/larray, to map a bigger file, which uses JNI to map files > 2GB, and provides it's own Buffer abstraction, using long data pointers.

One more option is to use Unsafe operations, like described here.

zeppelin
  • 8,947
  • 2
  • 24
  • 30
2

JDK14 offers a new solution to this limitation of the FileChannel API with the introduction of the Foreign-Memory Access API (JEP 383) preview. See the MemorySegment.mapFromPath() method to get started.

fralalonde
  • 151
  • 4