2

I am loading a big text file into memory (filling a class with a table to hold all the data). Text files range from several MB to 1 GB. However when loading a text file around 100 MB my java app running on windows 7 x64 basically just stops in the middle of reading the text file at about 40%. I am checking for free memory as I load each line.

The last output from println before the code stops reading the text file is

Total Memory: 688128 Free Memory: 86032 Used Memory: 602095

In task manager java.exe is consuming about 838 MB of Ram.

Anyway I can page memory to/from disk or increase the max memory? I will be running the application on multiple systems.

trincot
  • 317,000
  • 35
  • 244
  • 286
user2020457
  • 165
  • 1
  • 1
  • 13
  • Could your system be [thrashing](http://en.wikipedia.org/wiki/Thrashing_%28computer_science%29)? Continuously trying to find more memory, and paging things to disk or doing a full GC that only releases a small amount of memory before the system has to start looking for memory again. Running with `-verbose:gc` should tell you if the GC is thrashing and `top`/Windows process manager should tell you about excess paging. – Mike Samuel Feb 20 '13 at 00:38
  • You might be better off with a database such as MongoDB, Redis or even a distributed ehcache or gridgain. – beny23 Feb 20 '13 at 00:42
  • Consider using `Stream`s and `Reader`s, rather than loading the whole file into JVM. You have to try to process the file on-the-fly. – gaborsch Feb 20 '13 at 00:53
  • Concerning the current situation: try running `jstat` with different parameters, e.g. `jstat -gcutil`. Here http://docs.oracle.com/javase/1.5.0/docs/tooldocs/share/jstat.html you can read the different options. – gaborsch Feb 20 '13 at 00:58
  • 2
    Also would be interesting to know the JVM startup parameters (specifically the memory options) – gaborsch Feb 20 '13 at 01:00
  • If you cannot increase the JVM's startup parameters, your options would be: either to decrease the storage you need when you read it in, or to push some of it back to disk. If you do the latter, you should probably look into a simple Java caching library that does most of the work for you. – Darius X. Feb 20 '13 at 01:34
  • Which is the JVM command line? – fglez Mar 04 '13 at 11:40
  • Possible duplicate of [XX:+HeapDumpOnOutOfMemoryError Max file size limit](http://stackoverflow.com/questions/35930340/xxheapdumponoutofmemoryerror-max-file-size-limit) – Paul Sweatte Mar 07 '17 at 21:17

0 Answers0