1

I've been having a tough time memory mapping a 550MB file. I understand that 32-bit JVM can allocate a maximum memory size of around 1.4G, so I need to map a large file by parts. However, this is a 550MB file but I still cannot map it all into memory. The best I can do is a buffer size of around 333MB.

Below is my test code:

void testMap() throws IOException{
    long buffer = 500000000; // CAUSES ERROR. best I can do is 350000000
    RandomAccessFile srcFile = new RandomAccessFile("550MBFile", "r");
    ByteBuffer srcbb = srcFile.getChannel().map(MapMode.READ_ONLY, 0, buffer);
}

And the error is as following:

Exception in thread "main" java.io.IOException: Map failed at sun.nio.ch.FileChannelImpl.map(Unknown Source) at TestSpliter.testMap(TestSpliter.java:22) at TestSpliter.main(TestSpliter.java:14) Caused by: java.lang.OutOfMemoryError: Map failed ...

JVM argument: -Xms1024m

Can anyone explain why I can only use 300MB out of 1.4GB? Thanks.

Note that this is not a duplicate question with those ones asking about the maximum heap size for 32-bit JVM.

Fenwick
  • 1,061
  • 2
  • 15
  • 28
  • 1
    Why can't you use a 64-bit JVM? – user253751 Nov 09 '14 at 01:19
  • 1
    @immibis: Probably "because IT says so"... – Oliver Charlesworth Nov 09 '14 at 01:20
  • @immibis Just setting up a new Windows 8.1 PC and testing a couple old code. Java.com provides 32-bit download for Win8 by default, so I think it would be great if my application can support a 32-bit JVM. Actually I think the problem here is not realy about 32-bit, since the actual address space I get is much smaller than the predicted capability. – Fenwick Nov 09 '14 at 01:32

1 Answers1

2

Most likely, your address space is fragmented and there isn't more than 550MB available of contiguous address space. If you need to memory map large files, you need to be able to map them in segments if that becomes necessary.

David Schwartz
  • 179,497
  • 17
  • 214
  • 278
  • Thanks for answering. I'm not sure how exactly JVM allocates memory, but according to a couple of posts here on SO, JVM allocates a large contiguous region of memory of the maximum size. Doesn't this mean Java programs are guaranteed to have a region of contiguous address space that is approximately the size of JVM heap size? In other words, if JVM can be initialized, there will probably be enough contiguous address space? – Fenwick Nov 09 '14 at 02:03
  • @Fenwick It depends on the platform. For example, on some platforms, libraries are loaded at a fixed address if that address is available, to improve page sharing and minimize relocations. That can mean that memory is badly fragmented just when the process finally gets going. Migrate to 64-bits if at all possible. – David Schwartz Nov 09 '14 at 02:32
  • 2
    @Fenwick If the JVM can be initialized, and it allocates a 1GB heap, then there *was* 1GB of contiguous unused space. But that 1GB isn't unused any more, because the heap is using it. – user253751 Nov 09 '14 at 04:31