Java process doesn't exit on SIGTERM under heavy load

Question

In normal operation my application exits fine when sent a 'kill -s SIGTERM '.

However, under load sometimes the process does not exit.

I'm just wondering if it possible that http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6392332 is the reason for this, or if it could be something else?

Here are some parts of the stack trace of the process in question, showing the shutdown methods, any help is much appreciated.

Note, this is a Java process running on 64-bit RHEL 6.3.

2013-05-22 08:01:33 
Full thread dump Java HotSpot(TM) 64-Bit Server VM (23.21-b01 mixed mode): 

...

"Thread-15" prio=10 tid=0x000000001994d000 nid=0x4d5a waiting on condition [0x00007f4da08a3000] 
   java.lang.Thread.State: TIMED_WAITING (parking) 
at sun.misc.Unsafe.park(Native Method) 
- parking to wait for <0x000000079fd9fce8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) 
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) 
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) 
at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1468) 
at org.jboss.netty.util.internal.ExecutorUtil.terminate(ExecutorUtil.java:109) 
at org.jboss.netty.util.internal.ExecutorUtil.terminate(ExecutorUtil.java:49) 
at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.releaseExternalResources(AbstractNioWorkerPool.java:77) 
at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.releaseExternalResources(NioServerSocketChannelFactory.java:164) 
at com.test.services.radius.server.RadiusServerImpl.stop(RadiusServerImpl.java:87) 
at com.test.services.radius.ServiceProvider.unload(ServiceProvider.java:61) 
at com.test.spf.ServiceProviderCacheImpl.clearCurrentCache(ServiceProviderCacheImpl.java:150) 
- locked <0x00000006b8968038> (a com.test.spf.ServiceProviderCacheImpl) 
at com.test.spf.ServiceProviderCacheImpl.unload(ServiceProviderCacheImpl.java:170) 
at com.test.spf.SPAImpl.stop(SPAImpl.java:178) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:601) 
at org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:273) 
at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:199) 
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:487) 
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:463) 
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:480) 
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:463) 
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:480) 
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:463) 
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:480) 
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:463) 
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:431) 
- locked <0x00000006da966290> (a java.util.LinkedHashMap) 
at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1048) 
at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1022) 
at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:970) 
- locked <0x000000073ad1a920> (a java.lang.Object) 
at org.springframework.web.context.ContextLoader.closeWebApplicationContext(ContextLoader.java:384) 
at org.springframework.web.context.ContextLoaderListener.contextDestroyed(ContextLoaderListener.java:78) 
at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:4245) 
at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4886) 
- locked <0x00000006b8968118> (a org.apache.catalina.core.StandardContext) 
at org.apache.catalina.core.ContainerBase.removeChild(ContainerBase.java:936) 
at org.apache.catalina.startup.HostConfig.undeployApps(HostConfig.java:1359) 
at org.apache.catalina.startup.HostConfig.stop(HostConfig.java:1330) 
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:326) 
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142) 
at org.apache.catalina.core.ContainerBase.stop(ContainerBase.java:1098) 
- locked <0x00000006b89682f8> (a org.apache.catalina.core.StandardHost) 
at org.apache.catalina.core.ContainerBase.stop(ContainerBase.java:1110) 
- locked <0x000000068e63ff10> (a org.apache.catalina.core.StandardEngine) 
at org.apache.catalina.core.StandardEngine.stop(StandardEngine.java:468) 
at org.apache.catalina.core.StandardService.stop(StandardService.java:604) 
- locked <0x000000068e63ff10> (a org.apache.catalina.core.StandardEngine) 
at org.apache.catalina.core.StandardServer.stop(StandardServer.java:788) 
at org.apache.catalina.startup.Catalina.stop(Catalina.java:662) 
at org.apache.catalina.startup.Catalina$CatalinaShutdownHook.run(Catalina.java:706) 

"SIGTERM handler" daemon prio=10 tid=0x0000000025453000 nid=0x4d58 in Object.wait() [0x00007f4da0b2f000] 
   java.lang.Thread.State: WAITING (on object monitor) 
at java.lang.Object.wait(Native Method) 
- waiting on <0x000000072c10db20> (a org.apache.catalina.startup.Catalina$CatalinaShutdownHook) 
at java.lang.Thread.join(Thread.java:1258) 
- locked <0x000000072c10db20> (a org.apache.catalina.startup.Catalina$CatalinaShutdownHook) 
at java.lang.Thread.join(Thread.java:1332) 
at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:106) 
at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46) 
at java.lang.Shutdown.runHooks(Shutdown.java:123) 
at java.lang.Shutdown.sequence(Shutdown.java:167) 
at java.lang.Shutdown.exit(Shutdown.java:212) 
- locked <0x0000000707855738> (a java.lang.Class for java.lang.Shutdown) 
at java.lang.Terminator$1.handle(Terminator.java:52) 
at sun.misc.Signal$1.run(Signal.java:212) 
at java.lang.Thread.run(Thread.java:722)

@JIV would really rather get to the root cause of why SIGTERM isn't working here, instead of just using SIGKILL — user1977749, May 23 '13 at 12:23
its simple, SIGTERM notify application to terminate itself, its busy and cannot process it OR waiting to shutdown all thread correctly (which depends how app is written - can swallow interrupt and never exit). SIGKILL terminate it for you. — JIV, May 23 '13 at 13:11

score 1 · Answer 1 · answered May 25 '13 at 22:33

No Unix process will finish on SIGTERM till all threads are finished properly, thus returned from run method. High load could have its roots in deadlock - deadlocked threads usually never ends. Or endless loop in some thread.

Btw proper way of shutting down the Tomcat is to use bundled shutdown script. It could fail in some extraordinary situations, but then you can just SIGKILL it.

SIGTERM or SIGKILL, that's often the business question. Proper shutdown can easily take more than 15min, when complex application, swapped... a.s.o. So can you stand 15min outage or rather you'll kill it and start in next 2min?

Java process doesn't exit on SIGTERM under heavy load

1 Answers1