6

I am debugging a problem that I've had for years in a Tomcat application - a memory leak caused when restarting an application since the Webapp classloader cannot be GC'd. I've taking snapshots of the heap with JProfiler and it seems that at least some my static variables aren't getting freed up.

Certain classes have a static final member which is initialized when the class is first loaded, and because it's final I can't set it to null on app shutdown.

Are static final variables an anti-pattern in Tomcat, or am I missing something? I've just starting poking around with JProfiler 8 so I may be misinterpreting what the incoming references are telling me.

Cheers!

Luke

Luke Kolin
  • 115
  • 2
  • 9

3 Answers3

6

It is from a few years ago but this presentation I gave at JavaOne covers this topic exactly. The key steps to find the leak are in slide 11 but there is a lot of background information that might be useful as well.

The short version is:

  • Trigger the leak
  • Force GC
  • Use a profiler to find an instance of org.apache.catalina.loader.WebappClassLoader that has the property started=false
  • Trace the GC roots of that object - those are your leaks

As I note in the presentation, finding the leaks is one thing, finding what triggered them can be a lot harder.

I would recommend running on the latest stable Tomcat version as we are always improving the memory leak detection and prevention code and the warnings and errors that that generates may also provide some pointers.

Mark Thomas
  • 16,339
  • 1
  • 39
  • 60
  • I'm on 7.0.47 and unfortunately the gc roots of the class loader are several hundred. There are only about a dozen instances live and they are all static variables of classes - with the exception of Log4j 1.x classes. Perhaps those are what's keeping the classloader alive, and therefore keeping the statics around? – Luke Kolin Oct 30 '13 at 03:48
  • Sounds very plausible. Are you calling LogManager.shutdown() from a ServletContextLister.destroy() method to clean up log4j? – Mark Thomas Oct 30 '13 at 19:44
  • I was, but there still were a few log4j classes left lying around. I refactored my app to use a tomcat lifecycle listener to load log4j and moved the log4j jar into $TOMCAT/lib so the webapp classloaders would not load it. All that's left after app shutdown is my app, and tracing the GC roots just links to statics and class files. :( – Luke Kolin Nov 01 '13 at 15:37
  • 2
    Found it. sun.awt.AWTAppContext - need to ensure that the context is being grabbed by something above the Webapp classloader. http://cdivilly.wordpress.com/2012/04/23/permgen-memory-leak/ – Luke Kolin Nov 01 '13 at 16:08
  • You need to be careful with that sun.awt.AppContext. Changes in 1.7.0_25 onwards mean that calling that method requires a graphical environment and starts a AWT thread. Tomcat disabled this protection by default because of these changes (and that the original problem triggering the leak was fixed in 1.7.0_02 onwards) – Mark Thomas Nov 01 '13 at 21:58
  • @Mark Thomas. In your presentation there is statement "These references often remain after a web application reload". Can you explain what do you mean by reload here. is it restart of webserver? I think with restart , previous permgen memory gets cleared. Is n't it? – M Sach Nov 07 '13 at 05:31
  • There is no web server, we are discussing a Servlet container. reload does not mean restarting the entire container. reload means stop the web application and start it again. Obviously, if you stop the entire process and start a new one then PermGen will be cleared. – Mark Thomas Nov 08 '13 at 09:35
5

Static variables should be garbage collected when the class itself is garbage collected, which in turn is the case when its class loader is garbage collected.

You can easily create memory leak by having anything that wasn't loaded by the applications classloader having a reference to any of your classes (or an instance of your classes). Look for things like callback listeners etc. that you didn't remove properly (inner/anonymous classes are easily overlooked).

A single reference to one of your classes prevents its class loader and in turn any class loaded by that class loader to be garbage collected.

Edit, example of leaking an object that prevents GC of all your classes:

MemoryMXBean mx = ManagementFactory.getMemoryMXBean();
NotificationListener nl = new NotificationListener() { ... };
((NotificationEmitter) mx).addNotificationListener(nl, ..., ...);

If you register a listener (NotificationListener here) with an object that exists ouside of your applications scope (MemoryMXBean here), your listener will stay 'live' until its explicitly removed. Since your listener instance hold a reference to its ClassLoader (your application classloader) you have now created a strong reference chain preventing GC of the classloader, and in turn, all the classes it loaded, and through that, any static variables those classes hold.

Edit2: Basically you need to avoid this situation:

[Root ClassLoader]
       |
       v
      [Application ClassLoader]
               |
               v
               (Type loaded by Root).addSomething()

The JVM running the application server has loaded the JRE trough the root class loader (and possibly the application server, too). That means those classes will never become eligible for GC, since there will always be live references to some of them. The application server will load your application in a separate class loader that it will not hold a reference to any longer when your application is redeployed (or at least should). But your application will share all classes from at least the JRE with the application server (at least the JRE, but commonly also the Application Server).

In the hypothetical case when the application server were to create a separate class loader (with no parent, a second root class loader practically) and try to load the JRE a second time (as private to your application) it would cause a lot of problems. Classes intended to be singletons would exists twice, and the two class hierarchies would be incapable of holding any refrences of the other (Caused by the same class loaded by different class loaders beeing different types for the JVM). They couldn't even use java.lang.Object as a reference type for the respective "other" class loaders objects.

Durandal
  • 19,919
  • 4
  • 36
  • 70
  • Is there any way to easily find this using JProfiler? I've looked at the retained references and as best I can tell they're just static variables in the classes. What might be a clue is that they're classes loaded by the ServletContextListener, but it itself seems to have been collected. – Luke Kolin Oct 27 '13 at 23:18
  • @LukeKolin I'm not sure about what JProfiler exactly shows, but in principle you would simply look for classes from outside your application holding any reference to classes from your application. If everything you have is for example below the package com.mydomain, anything holding a reference com.mydomain, that isn't from com.mydomain is a candidate to be investigated. If you're lucky the name of the retained objects class will already tell you where it has been leaked. – Durandal Oct 28 '13 at 13:40
  • @Durandal can you give an example of statement "You can easily create memory leak by having anything that wasn't loaded by the applications classloader having a reference to any of your classes (or an instance of your classes)." I am just wondering when that is possible and which classloader loads the class if not application classloader? – M Sach Oct 29 '13 at 06:27
  • @MSach Please check my edit, I hope that makes things clearer. – Durandal Oct 29 '13 at 17:52
  • Thanks Durandal. I am close . But still i have a question Agreed MemoryMXBean object is remote object and actually exist somewhere else which internally hold the reference to NotificationListener.But my question is when GC is run on my applycation wont it collect the NotificationListener reference also becoz it wont find any further reference of that listener with in application and it can not search the reference on remote application. so should not it be Garbage collected> – M Sach Oct 30 '13 at 05:25
  • @MSach No MemoryMXBean is a *local* object existing in the context of the root ClassLoader (I specifically chose this one because I know its implemented as a simple static variable, essentially its a singleton in the JRE). That means in this case there is a strong reference chain from the root classloader to the listener. You can create the very same effect with any class of the JRE that has some static container of listeners. Runtime.addShutdownHook(Thread) is another such example (there is a camouflaged reference to your Runnable inside that thread). – Durandal Oct 30 '13 at 13:45
  • Its somewhat hard for me to come up with a good examples, most blunders in that regard will never be detectable in a desktop environment (where I mainly work). I'm sure any J2EE veteran could give you much better examples where this problem situation arises. – Durandal Oct 30 '13 at 13:49
1

This Blog can give you an idea about the memory leak in your application.

Debojit Saikia
  • 10,532
  • 3
  • 35
  • 46
  • While that's helpful, I'm not making references to external code. This is just code within my webapp, all loaded by the same classloader. – Luke Kolin Oct 27 '13 at 16:36