jstack 100% CPU

Question

So I got stuck trying to figure out why our application will occasionally consume 100% CPU. I've read several very helpful stackovertflow post, but I think to be on the safe side I would like to get some advice...guidance... assistance...whatever. Really what this comes down too is my inexperience looking through stack traces. I would really appreciate if someone could point out the possible culprit here as well as show me so pointers so I can better understand this.

The noteworthy threads (so it would seem to me) are ones waiting for Monitors:

I get four of these threads consistently jndi-0 - jndi-4

    "jndi-0" prio=6 tid=0x000000001113e000 nid=0x187c waiting for monitor entry [0x0000000012a9f000]
       java.lang.Thread.State: BLOCKED (on object monitor)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:388)
        - waiting to lock  (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at jrun.servlet.network.NetworkService.accept(NetworkService.java:356)
        at jrun.naming.NamingNetworkService.accept(NamingNetworkService.java:115)
        at jrun.naming.NamingNetworkService.createRunnable(NamingNetworkService.java:141)
        at jrunx.scheduler.ThreadPool$DownstreamMetrics.createRunnable(ThreadPool.java:287)
        at jrunx.scheduler.ThreadPool$ThreadThrottle.createRunnable(ThreadPool.java:349)
        at jrunx.scheduler.ThreadPool$UpstreamMetrics.createRunnable(ThreadPool.java:241)
        at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)
       Locked ownable synchronizers:
        - None


    "jrpp-298" prio=6 tid=0x0000000014730800 nid=0xdd4 waiting for monitor entry [0x000000001cc4b000]
       java.lang.Thread.State: BLOCKED (on object monitor)
        at coldfusion.runtime.LocalScope.put(LocalScope.java:247)
        - waiting to lock  (a coldfusion.runtime.VariableScope)
        at coldfusion.runtime.NeoPageContext.SymTab_setValidatedSimpleName(NeoPageContext.java:1339)
        at coldfusion.runtime.NeoPageContext.SymTab_setValidatedDottedName(NeoPageContext.java:1297)
        at coldfusion.runtime.NeoPageContext.setValidatedAttribute(NeoPageContext.java:524)
        at coldfusion.runtime.NeoPageContext.setAttribute(NeoPageContext.java:392)
        at coldfusion.runtime.CfJspPage._set(CfJspPage.java:369)
        at cfmathFunctions2ecfm1090027769$funcREDUCEBARBASE.runFunction(C:\SIWEB\cfw\controllers\ContentLogic\mathFunctions.cfm:711)
        at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:418)
        at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:324)
        at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:59)
        at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:277)
        at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:192)
        at coldfusion.runtime.CfJspPage._invokeUDF(CfJspPage.java:2471)
        at Statement26069.evaluate( :1)
        at coldfusion.compiler.ExprClassLoader.evaluate(ExprClassLoader.java:143)
        at coldfusion.runtime.CFPage.Evaluate(CFPage.java:1726)
        at cfContentLogic2ecfc1938419849$funcEVALUATEFUNCTIONSTRING.runFunction(C:\SIWEB\cfw\controllers\ContentLogic.cfc:479)
        at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:418)
        at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:324)
        at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:59)
        at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:277)
        at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:463)
        at coldfusion.runtime.CfJspPage._invokeUDF(CfJspPage.java:2436)
        at cfContentLogic2ecfc1938419849$funcFINDANDEVALUATE.runFunction(C:\SIWEB\cfw\controllers\ContentLogic.cfc:528)
        at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:418)
        at coldfusion.filter.SilentFilter.invoke(SilentFilter.java:47)
        at coldfusion.runtime.UDFMethod$ReturnTypeFilter.invoke(UDFMethod.java:360)
        at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:324)
        at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:59)
        at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:277)
        at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:192)
        at coldfusion.runtime.CfJspPage._invokeUDF(CfJspPage.java:2471)
        at cfContentLogic2ecfc1938419849$funcPREPAREDANSWERSBYQUESTIONIDQGID.runFunction(C:\SIWEB\cfw\controllers\ContentLogic.cfc:340)
        at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:418)
        at coldfusion.runtime.UDFMethod$ReturnTypeFilter.invoke(UDFMethod.java:360)
        at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:324)
        at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:59)
        at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:277)
        at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:463)
        at coldfusion.runtime.CfJspPage._invokeUDF(CfJspPage.java:2436)
        at cfContentLogic2ecfc1938419849$funcPROCESSDYNAMICITEMBYQUESTIONIDQGID.runFunction(C:\SIWEB\cfw\controllers\ContentLogic.cfc:69)
        at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:418)
        at coldfusion.runtime.UDFMethod$ReturnTypeFilter.invoke(UDFMethod.java:360)
        at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:324)
        at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:59)
        at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:277)
        at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:463)
        at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:453)
        at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:320)
        at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:2222)
        at cfitemPackaging2ecfm1230617345$funcPREPAREDITEMBYQUESTIONIDQGID.runFunction(C:\SIWEB\cfw\controllers\ProtectedBusinessLogic\itemPackaging.cfm:693)
        at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:418)
        at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:324)
        at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:59)
        at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:277)
        at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:463)
        at coldfusion.runtime.CfJspPage._invokeUDF(CfJspPage.java:2436)
        at cfitemPackaging2ecfm1230617345$funcPREPAREDITEMSANDPASSAGESSTRUCTBYQGID.runFunction(C:\SIWEB\cfw\controllers\ProtectedBusinessLogic\itemPackaging.cfm:408)
        at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:418)
        at coldfusion.runtime.UDFMethod$ReturnTypeFilter.invoke(UDFMethod.java:360)
        at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:324)
        at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:59)
        at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:277)
        at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:463)
        at coldfusion.runtime.CfJspPage._invokeUDF(CfJspPage.java:2436)
        at cfitemPackaging2ecfm1230617345$funcPREPAREDITEMSANDPASSAGESSTRUCTBYQGIDS.runFunction(C:\SIWEB\cfw\controllers\ProtectedBusinessLogic\itemPackaging.cfm:73)
        at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:418)
        at coldfusion.runtime.UDFMethod$ReturnTypeFilter.invoke(UDFMethod.java:360)
        at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:324)
        at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:59)
        at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:277)
        at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:463)
        at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:453)
        at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:320)
        at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:2222)
        at cfpracticesession2ecfm1218050444$funcPRACTICESESSION.runFunction(C:\SIWEB\cfw\controllers\Test\practicesession.cfm:272)
        at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:418)
        at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:324)
        at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:59)
        at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:277)
        at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:463)
        at coldfusion.runtime.CfJspPage._invokeUDF(CfJspPage.java:2436)
        at coldfusion.tagext.lang.InvokeTag.doEndTag(InvokeTag.java:432)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2662)
        at cfcfml2ecfm667848367$func$INVOKE.runFunction(C:\SIWEB\cfw\wheels\global\cfml.cfm:177)
        at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:418)
        at coldfusion.filter.SilentFilter.invoke(SilentFilter.java:47)
        at coldfusion.runtime.UDFMethod$ReturnTypeFilter.invoke(UDFMethod.java:360)
        at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:324)
        at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:59)
        at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:277)
        at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:463)
        at coldfusion.runtime.CfJspPage._invokeUDF(CfJspPage.java:2436)
        at cfprocessing2ecfm76711320$func$CALLACTION.runFunction(C:\SIWEB\cfw\wheels\controller\processing.cfm:85)
        at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:418)
        at coldfusion.filter.SilentFilter.invoke(SilentFilter.java:47)
        at coldfusion.runtime.UDFMethod$ReturnTypeFilter.invoke(UDFMethod.java:360)
        at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:324)
        at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:59)
        at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:277)
        at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:463)
        at coldfusion.runtime.CfJspPage._invokeUDF(CfJspPage.java:2436)
        at cfprocessing2ecfm76711320$func$PROCESSACTION.runFunction(C:\SIWEB\cfw\wheels\controller\processing.cfm:60)
        at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:418)
        at coldfusion.filter.SilentFilter.invoke(SilentFilter.java:47)
        at coldfusion.runtime.UDFMethod$ReturnTypeFilter.invoke(UDFMethod.java:360)
        at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:324)
        at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:59)
        at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:277)
        at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:192)
        at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:448)
        at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:308)
        at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:2272)
        at cfrequest2ecfm104933708$func$REQUEST.runFunction(C:\SIWEB\cfw\wheels\dispatch\request.cfm:166)
        at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:418)
        at coldfusion.filter.SilentFilter.invoke(SilentFilter.java:47)
        at coldfusion.runtime.UDFMethod$ReturnTypeFilter.invoke(UDFMethod.java:360)
        at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:324)
        at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:59)
        at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:277)
        at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:192)
        at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:448)
        at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:308)
        at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:2272)
        at cfindex2ecfm812783003.runPage(C:\SIWEB\cfw\wheels\index.cfm:1)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cfrewrite2ecfm453288076.runPage(C:\SIWEB\cfw\rewrite.cfm:1)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cfonrequest2ecfm951311187$funcONREQUEST.runFunction(C:\SIWEB\cfw\wheels\events\onrequest.cfm:1)
        at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:418)
        at coldfusion.runtime.UDFMethod$ReturnTypeFilter.invoke(UDFMethod.java:360)
        at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:324)
        at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:59)
        at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:277)
        at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:192)
        at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:448)
        at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:308)
        at coldfusion.runtime.AppEventInvoker.invoke(AppEventInvoker.java:74)
        at coldfusion.runtime.AppEventInvoker.onRequest(AppEventInvoker.java:243)
        at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:269)
        at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)
        at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
        at coldfusion.filter.PathFilter.invoke(PathFilter.java:86)
        at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)
        at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
        at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
        at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)
        at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
        at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
        at coldfusion.filter.RequestThrottleFilter.invoke(RequestThrottleFilter.java:126)
        at coldfusion.CfmServlet.service(CfmServlet.java:175)
        at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
        at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
        at com.intergral.fusionreactor.filter.FusionReactorFilter.k(FusionReactorFilter.java:636)
        at com.intergral.fusionreactor.filter.FusionReactorFilter.d(FusionReactorFilter.java:266)
        at com.intergral.fusionreactor.filter.FusionReactorFilter.doFilter(FusionReactorFilter.java:171)
        at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
        at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)
        at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
        at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
        at jrun.servlet.FilterChain.service(FilterChain.java:101)
        at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
        at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
        at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)
        at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
        at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
        at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
        at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
        at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
        at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)
       Locked ownable synchronizers:
        - None


    Trace Time:   10:50:51.504 18-Sep-2012Thread ID:    0x19 (25)Thread Name:  jndi-3Priority:     5Hashcode:     290417686State:        RUNNABLE"jndi-3" prio=5 runnablejava.net.PlainSocketImpl.socketAccept(PlainSocketImpl.java:???)[Native Method]java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)java.net.ServerSocket.implAccept(ServerSocket.java:453)java.net.ServerSocket.accept(ServerSocket.java:421)jrun.servlet.network.NetworkService.accept(NetworkService.java:356)jrun.naming.NamingNetworkService.accept(NamingNetworkService.java:115)jrun.naming.NamingNetworkService.createRunnable(NamingNetworkService.java:141)jrunx.scheduler.ThreadPool$DownstreamMetrics.createRunnable(ThreadPool.java:287)jrunx.scheduler.ThreadPool$ThreadThrottle.createRunnable(ThreadPool.java:349)jrunx.scheduler.ThreadPool$UpstreamMetrics.createRunnable(ThreadPool.java:241)jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

These threads are BLOCKED which means they should be using any CPU. Can you find RUNNABLE threads? — Peter Lawrey, Sep 17 '12 at 19:52
There is probably a thread locked and causing the others to block. And because it doesnt unlock the monitor can never continue. I guess. — fonZ, Sep 17 '12 at 19:55
> The noteworthy threads (so it would seem to me) are ones waiting for > Monitors: Actually these are not noteworthy since they are in `WAITING` state and consume no CPU. You should inspect the `RUNNING` threads. Either some thread does something that is CPU-intensive (is that bad? can you improve it? *you* should know) or another that comes to mind is a live-lock. Your threads content for the CPU but no real progress is done. If your process finishes its job but just consumes too much CPU you can exclude the live-lock though. — Cratylus, Sep 17 '12 at 20:50
The stack traces you posted are of `Thread`s which are blocked, so not consuming any CPU at the moment. The easiest way to discover which `Thread` (or `Thread`s) are using all you CPU is attaching a profiler to your application: examples of such profilers are JProfiler or even the freely available JVisualVM. You can also use some command-line utilities to discover the stack-trace of the `Thread` eating your CPU. Take a look at [this blogpost](http://nurkiewicz.blogspot.be/2012/08/which-java-thread-consumes-my-cpu.html) for an example — Robin, Sep 17 '12 at 20:32
So I am pretty sure this is the thread (last thread I posted) that is causing the blocks — user1678402, Sep 18 '12 at 15:55

score 0 · Answer 1 · answered Oct 04 '12 at 21:04

0

I'll second the recommendation to try JVisualVM - it is great for sorting through threads. Another good option is to use Eclipse, which will let you step through individual threads in debug mode as well.

answered Oct 04 '12 at 21:04

Chuck Mosher

985
2
8
11

jstack 100% CPU

1 Answers1