0

In production environment I got frequently thread hung error message.

Log is below.

            [12/16/15 16:47:30:095 PHT] 00000030 ThreadMonitor W   WSVR0605W: Thread "WebContainer : 11" (000005bb) has been active for 743576 milliseconds and may be hung.  There is/are 5 thread(s) in total in the server that may be hung. 
                    at java.util.HashMap.getEntry(HashMap.java:479) 
                    at java.util.HashMap.get(HashMap.java:429) 
                    at com.c2lbiz.symbiosys.insurance.product.business.ProductBO.getElementKeyString(ProductBO.java:222) 
                    at com.c2lbiz.symbiosys.insurance.product.business.ProductBO.evaluateField(ProductBO.java:2156) 
                    at com.c2lbiz.symbiosys.insurance.product.business.ProductBO.calculateFundFacets(ProductBO.java:2594) 
                    at com.c2lbiz.symbiosys.insurance.product.business.ProductBO.calculateIllustration(ProductBO.java:1113) 
                    at com.c2lbiz.symbiosys.insurance.quotation.util.QuotationUtil.generateIllustrationOutput(QuotationUtil.java:486) 
                    at com.c2lbiz.symbiosys.insurance.quotation.business.QuoteIllustration.execute(QuoteIllustration.java:49) 
                    at com.c2lbiz.symbiosys.insurance.controller.quotation.QuoteCaptureController.generateIllustrationPDF(QuoteCaptureController.java:176) 
                    at com.c2lbiz.symbiosys.insurance.controller.quotation.GetOutputPreferencesController.process(GetOutputPreferencesController.java:66) 
                    at com.c2lbiz.symbiosys.insurance.controller.quotation.QuoteCaptureController.process(QuoteCaptureController.java:1) 
                    at com.c2lbiz.symbiosys.systemcommon.controller.BaseController.handleRequestInternal(BaseController.java:106) 
                    at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153) 
                    at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48) 
                    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:900) 
                    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:827) 
                    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:874) 
                    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:790) 
                    at javax.servlet.http.HttpServlet.service(HttpServlet.java:595) 
                    at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) 
                    at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1230) 
                    at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:779) 
                    at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:478) 
                    at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178) 
                    at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136) 
                    at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:97) 
                    at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:359) 
                    at org.springframework.security.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109) 
                    at org.springframework.security.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83) 
                    at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371) 
                    at org.springframework.security.ui.ExceptionTranslationFilter.doFilterHttp(ExceptionTranslationFilter.java:101) 
                    at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53) 
                    at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371) 
                    at org.springframework.security.ui.AbstractProcessingFilter.doFilterHttp(AbstractProcessingFilter.java:271) 
                    at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53) 
                    at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371) 
                    at org.springframework.security.ui.logout.LogoutFilter.doFilterHttp(LogoutFilter.java:89) 
                    at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53) 
                    at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371) 
                    at org.springframework.security.ui.logout.LogoutFilter.doFilterHttp(LogoutFilter.java:89) 
                    at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53) 
                    at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371) 
                    at org.springframework.security.context.HttpSessionContextIntegrationFilter.doFilterHttp(HttpSessionContextIntegrationFilter.java:235) 
                    at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53) 
                    at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371) 
                    at org.springframework.security.concurrent.ConcurrentSessionFilter.doFilterHttp(ConcurrentSessionFilter.java:99) 
                    at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53) 
                    at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371) 
                    at org.springframework.security.util.FilterChainProxy.doFilter(FilterChainProxy.java:174) 
                    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) 
                    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259) 
                    at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195) 
                    at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91) 
                    at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:967) 
                    at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1107) 
                    at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:87) 
                    at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:919) 
                    at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662) 
                    at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:200) 
                    at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:463) 
                    at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:530) 
                    at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:316) 
                    at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:88) 
                    at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175) 
                    at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217) 
                    at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161) 
                    at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138) 
                    at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204) 
                    at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775) 
                    at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905) 
                    at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1881)

It started from hanging thread 1 onward means 

            There is/are 1 thread(s) in total in the server that may be hung. 
            There is/are 2 thread(s) in total in the server that may be hung. 
            There is/are 3 thread(s) in total in the server that may be hung. 
            There is/are 4 thread(s) in total in the server that may be hung. 
            There is/are 5 thread(s) in total in the server that may be hung. 
            There is/are 6 thread(s) in total in the server that may be hung. 
            There is/are 7 thread(s) in total in the server that may be hung. 
            There is/are 8 thread(s) in total in the server that may be hung. 
            There is/are 9 thread(s) in total in the server that may be hung. 
            There is/are 10 thread(s) in total in the server that may be hung. 
            ...............
            ...............
            There is/are 77 thread(s) in total in the server that may be hung.

After that PDF generation stops working and batch running also stops running. Batch job generate pdf and send to provided emailid. What is root cause of this error? Same code is working fine on UAT and SIT environment.

Brijesh
  • 148
  • 2
  • 16
  • How large is your hashmap? Is there an infinite loop of lookups? or any kind of code that syncronizes the lookup causing a deadlock? – Tommy Dec 21 '15 at 15:09
  • The size of Hashmap is not fixed, it's keep putting elements in map.I guess it become very big during execution.Is this is cause of thread hung?please suggest how to resolve this issue without code changing?The same code working fine in UAT and SIT environment. Application is deployed on websphere application server and backend database is oracle. – Brijesh Dec 22 '15 at 06:50

1 Answers1

2

Under some circumstances, a get() from an unsynchronized HashMap can cause an infinite loop. This can occur if a concurrent put() causes a rehash of the Map. In my case : I am using unsynchronized hashmap and concurrent thread calling get and put method on that hashmap. It caused infinite loop on get method and thread gets hung.It is my observation.

Brijesh
  • 148
  • 2
  • 16