0

I am running 1.6.0_25 64bit on windows 7 64bit.

I am trying to have the JVM run with maximum memory because my application is very memory intensive... but sadly... the memory allocation does not build up and there is a lot of Page Faults from windows to keep bringing the next set of virtual memory in.

I am running java -Xms2G -Xmx3G Test

The following code is my attempt at bringing a whole file in... such that i do not run into page faults for reading in.

File f = new File("veryLARGEfile.txt");
FileInputStream in = new FileInputStream(f);
int i = (int) f.length();
System.out.println("Bytes in file: " + i);

byte[] file = new byte[i];
i = in.read(file);
System.out.println("Bytes read: " + i);

doing this method i can see that in windows task manager the system reaches 2G worth of memory while it is reading the file... but once it is done reading it... the memory falls back down again!!!!

This is a major problem... i need the whole byte array to stay in active memory.

Thank you, ey

I have modified the code to use basic array types int[][] and float[][] to hold my data instead of keeping ArrayLists of an object containing int's and float's.

doing this, i find that my java memory does not get swapped (so, i guess heap memory is treated a bit differently from stack here) [oh, i did change all the code to be static typed as well - i know, very bad programming style]

the issue that i am running into now is how to handle my HashMap... all my attempts of trying to build a lookup table is failing with O(n^2) running time to build!!!

Bill the Lizard
  • 398,270
  • 210
  • 566
  • 880
Enas Yunis
  • 1
  • 1
  • So if I get this right, you're trying to purposefully create a memory leak? Or you need the file to be present in memory at all times? I am interested in knowing about your use case.. – lobster1234 Apr 25 '11 at 10:34
  • Do you maintain a reference to `byte[] file` after loading the file? How much memory does your system have? What does the rest of your code do with this file? – WhiteFang34 Apr 25 '11 at 10:37
  • Actually... I was originally reading a 1GB file using BufferReader and processing the input and outputing into the BufferedWriter... The system is supposed to create almost 5GB worth of processed data. The sad part is that watching the memory usage it is increasing extremely slowly from 30K to .5G in 6 hours and with every couple of k bytes increase another one or two IO reads are done... – Enas Yunis Apr 25 '11 at 10:41
  • that sounds like a use case where one would very much try to *avoid* having the entire file kept in memory (especially since it's quite easy to do that via the IO stream API). – Michael Borgwardt Apr 25 '11 at 10:43
  • continued... so i re-wrote the code to load the data quickly. So, great news, it does, it is loading in 2 minutes... but once the load is complete and i am ready to use the data, the memory goes back down, and then i start seeing very heavy page fualts. – Enas Yunis Apr 25 '11 at 10:46

2 Answers2

1

The decision what parts of a process's memory to keep in RAM and which to swap out is made by the OS, not by the JVM. Generally, the OS will keep memory that is accessed frequently in RAM - and if it isn't, why would you need it there?

There might be OS APIs that allow you to "pin" a certain piece of memory in RAM, but this functionality is not exposed by Java.

But for your requirements, you should look into having the file memory-mapped rather than reading it in explicitly. It will likely be much faster.

Michael Borgwardt
  • 342,105
  • 78
  • 482
  • 720
  • ok... so, how do i handle the fact that the data i am reading is ending up in ArrayLists and HashMaps with pre-set starting sizes this is the part that is being paged out... and the algorithm that i am implementing is very memory intensive... i was recommended to move to Linux... question here... will it behave any differently and allow me to keep the ArrayLists in active memory? – Enas Yunis Apr 25 '11 at 10:52
  • @Enas: Since paging is an OS affair, a different OS may well behave differently. Consumer editions of Windows tend to be optimzed for end-user applications rather than data processing. Moving to a Windows Server OS may also result in different paging behaviour. Though I'd say that once your algorithm works intensively with all of the data, it should get paged out at all on any sane OS. – Michael Borgwardt Apr 25 '11 at 11:31
  • Thank you, that makes a lot of sense. I am thinking of installing a Linux based on the system to see if that solves it... would you recommend Ubuntu or is there a better 'server' behaving Linux distribution. Thank you. – Enas Yunis Apr 25 '11 at 11:51
0

Java tries fairly hard to keep all its application in memory, or put another way, it swaps out as a last resort as it perform very badly if any of its memory is swapped to disk.

If your Java application is swapping to disk, its because your application uses too much memory for the system you have. If your application is memory intensive and you have to use this much memory I suggest sure your system has enough memory. BTW. You can buy a server with 8 GB for ~$500, and new 16 GB workstation for $1000. ;)

Peter Lawrey
  • 525,659
  • 79
  • 751
  • 1,130
  • I am running on a system with 24G of memory and 24G of virtual memory. I am planning now to eliminate the virtual memory as such forcing the system to use all the available physical memory... – Enas Yunis Apr 25 '11 at 15:45
  • Are you sure that this is what is causing the swapping, if you have 24 GB and you are using ~3 GB, I don't see why you would be swapping to virtual memory. There must be other things on your system causing this. – Peter Lawrey Apr 25 '11 at 20:59