4

I have a webapp that is subject to class loader leak(s), as PermGen runs out of memory after a number of re-deployments of the webapp.

Google and a couple of helpful persons on the web taught me that the way to investigate such an issue is to open a memory dump in a profiler, locate the faulty class loader, and look at the GC root paths that lead to this loader. Doing so with Yourkit, I could find a leak due to the CompositeClassLoader of Drools, and even managed to work around it. I think.

But after applying the fix, I still see the class loader not GC'ed after a redeploy. When I ask to Yourkit the paths to the loader, it can show literally hundreds of them. Please don't tell me this is the number of leaks I have to investigate!

Yourkit screenshot

(The first two paths are the ones from Drools, I took the snapshot before my fix. I also hid my company package names, just to avoid any complaint.)

And what's even more confusing, I don't get the same paths if I use the tab Paths to loader (Alt+3) than if I use the command Paths from GC roots (Ctrl+P) which opens a new tab:

enter image description here

So I would like to understand:

  • What does exactly this Paths to loader tab show? The list of classes that have a reference to this class loader? But then what is the difference with the Class list tab? And what is Merged paths? What is the difference between Paths to loader and Paths from GC roots? I am having trouble understanding precisely what these tabs show, and the documentation of Yourkit doesn't say much more than "The paths to loader tab shows the paths to the loader."...
  • More generally, how do I find the culprit(s) in this quantity of data?

I am mentioning Yourkit because this is what I use and am familiar with (at least for other profiling tasks) in case there is anything particular with it, but I would assume this is pretty much similar with other memory dump analyzers. The version of Yourkit I have is 8.0.

Djizeus
  • 4,161
  • 1
  • 24
  • 42
  • could you also post one or two full GC paths to the leaked classloader? also can you let us know how you are distinguishing the leaked classloader from the current one – Angular University Apr 07 '14 at 19:21
  • @jhadesdev To find the leaked loader I take a snapshot before redeploy, redeploy, force a GC and take another snapshot. Then I compare the two snapshots to see which one was present in both snapshots. I replaced the image to show expanded paths, and while doing so got another confusion between "Paths to loader" and "Paths from GC roots"... Please note that solving my problem would be nice, but what I am really looking at is understanding these features and get a generic methodology. – Djizeus Apr 08 '14 at 04:50
  • 1
    I think your method is correct, find the leaked classloader and find GC root paths that are keeping it hanging. This might a problem with user friendliness of the UI, consider giving a try with Eclipse Memory Analyser tool (MAT), and run the leak suspect report http://java.jiderhamn.se/2011/12/11/classloader-leaks-i-how-to-find-classloader-leaks-with-eclipse-memory-analyser-mat/, you will probably find more help online for MAT then for other profilers – Angular University Apr 08 '14 at 21:07
  • I posted a possible procedure with yourkit, the 'Paths to Loader' view is what is needed in this case – Angular University Apr 08 '14 at 21:32
  • @jhadesdev Thanks, yes I'll probably try MAT if I can't get to understand Yourkit, it seems widely used indeed. – Djizeus Apr 09 '14 at 04:04
  • in case you go with MAT, try to click the classloader and see the 'Immediate Dominators', the closest object in the referencer chain that holds the most memory, see here tip 4 of this post -> http://eclipsesource.com/blogs/2013/01/21/10-tips-for-using-the-eclipse-memory-analyzer/, this might help with such a large number of retaining paths – Angular University Apr 11 '14 at 19:50

1 Answers1

1

this is the way that Yourkit recommends to find the classloader leaks, they go through it here:

  • reproduce the problem with one redeploy, for a scenario of 2 classloaders, one leaked and one current
  • click on classloaders tab
  • Sort classloaders by 'number of classes without instances'
  • the leaked classloader will have lots of classes without instances, the instances have been collected but not the classes and the classloader itself, causing the permgen leak
  • once the leaked classloader is identified, select 'Paths to Loader'
  • the paths to the loader should be shown, there should be less than a handful

This is an example of a leak found:

classloader leak

This is the image that your are looking for, on top you have the leaked classloader, on the bottom you have the reason for the leak.

Usually the leak is caused by a static variable at a server classloader that holds references to objects on the leaked classloader (for example a JDBC driver shipped inside the WAR), or a thread started by the first classloader that outlived the deplyment and contains a thread local variable with a reference to an object of the ancient classloader.

Angular University
  • 42,341
  • 15
  • 74
  • 81
  • Yes I had seen this link, but _there should be less than a handful_ is not true for me, and that is what confuses me. Why does Yourkit show more than 100 paths? – Djizeus Apr 09 '14 at 04:01