8

I have written a java program which needs to process thousands of text files (all needs to be loaded on memory). It works fine with as many as 123 input files, but when I run it to process around 5000 files, it terminates unexpectedly in the middle of the road, without giving any error message/exception. Can anyone give me clue about what might have gone wrong?

I am using jdk1.6 on Mac OS Leopard having 2GB RAM.

user2864740
  • 60,010
  • 15
  • 145
  • 220
Fahim
  • 305
  • 2
  • 3
  • 14

5 Answers5

4

It seems like you are getting OutofmemoryError.

if it is the case try to increase heap memory size.

java -Xms<initial heap size> -Xmx<maximum heap size>
Upul Bandara
  • 5,973
  • 4
  • 37
  • 60
  • 1
    Yes, I have OutOfMemory error, which is coming from a recursive function that in-order traverse a tree of 136600 nodes. Is it due to the recursion? Should I try to remove recursion by loop? Is that possible? Yes, I can increase the heap size, which I consider to be the last option. – Fahim Feb 03 '10 at 16:43
  • how can I check the current heap size? Do I have to increase heap size every time before I run my program? – Fahim Feb 04 '10 at 03:47
  • You can use JConsole utility that comes with JDK. – Upul Bandara Feb 04 '10 at 12:36
  • thanks to all, it was a memory issue, as I discovered. And increasing the heapsize solved the problem. – Fahim Mar 10 '10 at 23:37
4

Given that it is your program, I suggest that you do the following:

First, change the main method so that everything is done in a try/catch block that reports all uncaught exceptions; e.g. something like this:

public static void main(String[] arghhhhh) {
    try {
        ...
    } catch (Throwable ex) {
        System.err.println("Uncaught exception - " + ex.getMessage());
        ex.printStackTrace(System.err);
    }
}

Second, look for anywhere that you might "squash" unexpected exceptions by catching them and not reporting them.

Third, look for anywhere that you might call System.exit() silently. This could happen in libraries too ... if you are using a badly written one.

If those measures don't give you answers, try to figure HOW the application is exiting by

  • by running from a debugger with breakpoints set at key points, or
  • by adding trace print statements at key points.
Stephen C
  • 698,415
  • 94
  • 811
  • 1,216
  • Yes, I have OutOfMemory error, which is coming from a recursive function that in-order traverse a tree of 136600 nodes. Is it due to the recursion? Should I try to remove recursion by loop? Is that possible? – Fahim Feb 03 '10 at 16:41
  • I doubt that recursion (of a directory tree of files) is causing this. You need to use a memory profiler to find out what is using the memory. – Stephen C Feb 03 '10 at 21:34
  • is there any chance i can see whats going on when i have the same error, but a try&catch arround main does not output anything? – reox Jun 12 '11 at 22:49
  • @reox - are you catching `Throwable`? – Stephen C Jun 13 '11 at 00:16
2

Are you opening files simultaneously? You might be running out of memory if you are loading too many files at once. If the files are large enough, you might be running out of memory with only a single file open. Also, make sure you are closing the files when you're done with them.

Andy White
  • 86,444
  • 48
  • 176
  • 211
  • Most of the files are 4 KB in size, and the largest one is 193 KB. All files' size sums up to 20 MB. I am reading those files one after another. I see that those are successfully read. Those files in fact provides input to my program. I don't make any change to the files. My program aborts in the middle of processing the inputs. If heap exhausted, I would expect memory exception. Since, I don't get any error message, I don't get any clue. – Fahim Feb 03 '10 at 06:30
  • * I closed all the files. * I haven't yet observed free memory with any tools? Can you suggest any tool? – Fahim Feb 03 '10 at 06:34
1

Check if you have any try/catch blocks that do not log exceptions properly.

It could most likely be OutofmemoryError. Make sure the console isn't redirected.

Vinodh Ramasubramanian
  • 3,157
  • 1
  • 20
  • 17
1

There are mainly two reasons.

  1. A "unhandled" system fault has occurred i.e. java.lang.OutOfMemoryError.
  2. A "unhandled" application fault has occurred.
  3. System.exit has been called.

To deal with these scenarios consider to do the following steps:

  • Look in you code for calls to System.exit.

  • Ensure that you handles all exceptions in the starting Java stack frame, i.e. Main-method:

    try{ ..code }catch(Throwable t){ t.printStackTrace }

  • Ensure that you have control where the stdout and stderr is directed. You may programatically set these to concrete files:

System.setOut(new PrintStream("output.txt")); System.setErr(new PrintStream("err.txt"));

  • Start with args as: -Xloggc:gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
Robert Höglund
  • 954
  • 9
  • 12