4

I have a complex Java application that runs for a long time. the application does the same operation, with little nuances, over and over again. My problem is that Windows task manager shows a lot of leaking handles for the java process (not file handles).

after some time of running the application in stress, I get strange failures, like the application getting stuck, dissappears, or I get FileNotFoundException when trying to open a file ("Insufficient system resources exist to complete the requested service").

I used Windows task manager to see what is the problem, and found that for the java.exe process - the number of handles is being increased very fast. the number of threads in not increased, and the amount of used RAM is also not increased.

I then used SysInternal's Process Explorer in order to understand what are these handles. I see that they are not file handles, but thousands of handles of type 'Mutant', with the name \BaseNamedObjects\??????n. According to some sites in the web, Mutant in Process explorer means Mutex. My Java app does not intentionally create any Mutexes.

The next step was to use profiling tools in order to narrow down the source of the leaks. I used "J Optimizer" & "Java VirtualVM". with both applications, I cannot detect the leaking handles. they have memory leak detectors, But I can't find a way to detect leaking handles.

My question is : How can I debug this problem ? How can I discover what causes the leaking handles ?

Thank you

amnong
  • 41
  • 1
  • 4
    If you use a lot of stream resources, ensure you open them right before a `try` and you close them in `finally`. But without code, this is difficult to guess. – fge Dec 25 '12 at 12:44
  • we need to see the code of what the operation is (or how it is started). Do you start a new thread for each operation. Do you use some kind of scheduling? Without code it would be nearly impossible to say what the problem is. – Nikola Yovchev Dec 25 '12 at 12:46
  • 2
    @baba then just suggest a way to _find_ the problem. Not always the question is what the problem _is_. – John Dvorak Dec 25 '12 at 12:47
  • @fge The question is more about how to find that particular code - which cannot always be that easy. – nos Dec 25 '12 at 12:53
  • 1
    @nos that is true... The problem however is that the tools used look for native leaks, and there is no one-to-one mapping between such resources and Java code. The moment when an `InputStream` falls out of scope, for instance, even when it is garbage collected, the underlying fd, if any, is not -- unless you have ensure it is closed. – fge Dec 25 '12 at 12:56
  • 1
    Can you give some more information about what your application does? Does it open sockets? If so, you could be leaking references to those. Does it create many short-lived threads? Those may successfully finish running, but could still be kept from being garbage collected in some way. In short: what resources does it use for its operation? – Confusion Dec 25 '12 at 12:56
  • my source code is huge so there is no way I can send it and for you to understand anything in it. @Confusion - It opens one socket for each iteration, so this doesn't explain the leaks. it opens a thread for each iteration, and also finishes the run() method. so as far as I know, there should not be any handles open. To make sure, I also explicitly call System.gc(), and see that the handles are not cleaned. – amnong Dec 27 '12 at 12:53
  • @fge your comment is interesting - you say that there is no way to map between the native leaks that task manager & process explorer show - and the leaking resources of Java ? Does anyone know of a method to map between them ? what kind of Java resources can cause native handles leaks ? – amnong Dec 27 '12 at 12:54
  • 1
    @amnong at the moment I can only think of two things, in order: 1. find, in all of your code, every created instance which implements `Closeable`, and ensure that all these instances do call `.close()` eventually (that means on error paths as well); 2. if 1. still leaks, with the help of data gathered from 1., interactively debug all paths... Yeah, undoable. So, other question: how do you test these code paths? – fge Dec 27 '12 at 15:02
  • @fge I am using SE 1.4.2. closeable is from SE 5.0 and above. Do you have a suggestion how to find a list of classes that should be reviewed ? – amnong Dec 27 '12 at 17:22
  • @amnong since all your code is 1.4.2 compatible (on the verge of 2013? Strange, but...), you can always set your working JDK to 1.5+ and apply the above. Of course, this supposes that ultimately, all the JDK-provided objects you use have been retrofitted to use it. And more to the point, why 1.4 in 2012+? – fge Dec 27 '12 at 17:38

1 Answers1

0

If you're curious about how the references which leak are allocated, see this: How to view memory allocation stacktrace in Java VisualVM

I'm not sure how you'd trace how references failed to deallocate.

Community
  • 1
  • 1
Daniel
  • 855
  • 10
  • 21