memory leak - metrics-meter-tick-thread and New I/O client worker

Question

I am running spring-mvc application. When i was closing Tomcat server, it shows

SEVERE: The web application [/myapp] appears to have started a thread named [metrics-meter-tick-thread-1] but has failed to stop it. This is very likely to create a memory leak.
SEVERE: The web application [/myapp] appears to have started a thread named [metrics-meter-tick-thread-2] but has failed to stop it. This is very likely to create a memory leak.

and this one :

SEVERE: The web application [/myapp] appears to have started a thread named [New I/O client worker #1-3] but has failed to stop it. This is very likely to create a memory leak.

SEVERE: The web application [/anant] created a ThreadLocal with key of type [org.jboss.netty.util.CharsetUtil$2] (value [org.jboss.netty.util.CharsetUtil$2@5db3978d]) and a value of type [java.util.IdentityHashMap] (value [{UTF-8=sun.nio.cs.UTF_8$Decoder@39a2da0a}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

this one don't know which jar is related with may be netty

when i explored jars dependency i saw that there are two metrics-core jars:

metrics-core:2.2.0 (used by `datastax`)
metrics-core:3.0.1 (used by `Titan`)

I am attaching all snaps to make it more clear. So what is the solution ????

I am using

jdk1.7 cassandra-driver-core-1.0.4 titan-0.4.4 cassandra-1.2.2 tomcat-7.0.34

Answer 1

I know I'm a bit late with this. I had the same issue and eventually found the solution. The problem is that the metrics 2.2 JAR, (which spawns these threads) uses the ManagementFactory.getPlatformMBeanServer() method, as suggested by Oracle. This class is in the java.lang package, so it will be loaded centrally by the VM and not for each module. So, since the Metrics package will only shut these threads down by itself on a VM exit (via adding a shutdown-hook), the classloader that loaded this package will let the MXBeans specified through the package linger on on the module's unload. What makes this even worse is that classloader that loaded the war file will also stay loaded on the VM, which (transitively) will also include any classes loaded in the module and any static state.

You can call Metrics.shutdown() manually, that sometimes solves the problem. I did have some exotic problems with that solution (sometimes the threads still stayed around after this, but I have a very peculiar setup and didn't want to waste any more time on this issue).

Answer 2

Check what started metrics-meter-tick-thread* threads , either your webapp or some library. And stop the thread before you shutdown application. See Tomcat wiki link explains how it creates a memory leak. Also it explains how uncleaned ThreadLocal variables causes memory leaks.

Threads spawned by webapps

If a webapp creates a thread, by default its context classloader is set to the one of the parent thread (the thread that created the new thread). In a webapp, this parent thread is one of tomcat worker threads, whose context classloader is set to the webapp classloader when it executes webapp code.

Furthermore, the spawned thread may be executing (or blocked in) some code that involves classes loaded by the webapp, thus preventing the webapp classloader from being collected.

So, if the spawned thread is not properly terminated when the application is stopped, the webapp classloader will leak because of the strong reference held by the spawned thread.