I'm using Java EntityManager to work with database in WSO2 AM and the application is working with several threads and sometimes the application is hanging when many requests comes to the application.
The thread dump is as follows,
Thread 23217: (state = BLOCKED)
- java.util.HashSet.add(java.lang.Object) @bci=8, line=217 (Compiled frame; information may be imprecise)
- java.util.AbstractCollection.addAll(java.util.Collection) @bci=29, line=342 (Compiled frame)
- java.util.LinkedHashSet.<init>(java.util.Collection) @bci=22, line=169 (Interpreted frame)
- org.eclipse.persistence.internal.jpa.metamodel.ManagedTypeImpl.getAttributes() @bci=13, line=160 (Interpreted frame)
- org.eclipse.persistence.internal.jpa.metamodel.IdentifiableTypeImpl.initializeIdAttributes() @bci=12, line=95 (Compiled frame)
- org.eclipse.persistence.internal.jpa.metamodel.MetamodelImpl.initialize(java.lang.ClassLoader) @bci=251, line=493 (Compiled frame)
- org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.getMetamodel(java.lang.ClassLoader) @bci=25, line=3476 (Interpreted frame)
- org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.deploy(java.lang.ClassLoader, java.util.Map) @bci=955, line=754 (Interpreted frame)
- org.eclipse.persistence.internal.jpa.EntityManagerFactoryDelegate.getAbstractSession() @bci=54, line=205 (Interpreted frame)
- org.eclipse.persistence.internal.jpa.EntityManagerFactoryDelegate.createEntityManagerImpl(java.util.Map, javax.persistence.SynchronizationType) @bci=5, line=305 (Interpreted frame)
- org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.createEntityManagerImpl(java.util.Map, javax.persistence.SynchronizationType) @bci=90, line=337 (Interpreted frame)
- org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.createEntityManager() @bci=3, line=303 (Interpreted frame)
- com.abc.example.dao.GenericDAO.<init>(com.abc.example..utils.DBProperties) @bci=111, line=35 (Interpreted frame)
- com.abc.example.dao.NewDAO.<init>(com.abc.example.utils.DBProperties) @bci=2, line=21 (Interpreted frame)
- com.abc.example.services.MyService.<init>(java.lang.String) @bci=56, line=19 (Interpreted frame)
- com.abc.example.services.InitService.<init>(com.abc.example.ws.GetRequest) @bci=3, line=17 (Interpreted frame)
- com.abc.example.ws.MySOAPImpl.getRequest(com.abc.example.ws.GetRequest) @bci=36, line=107 (Interpreted frame)
- sun.reflect.GeneratedMethodAccessor127.invoke(java.lang.Object, java.lang.Object[]) @bci=40 (Interpreted frame)
- sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Compiled frame)
- java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=57, line=606 (Compiled frame)
- org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(org.apache.cxf.message.Exchange, java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) @bci=57, line=181 (Interpreted frame)
- org.apache.cxf.service.invoker.AbstractInvoker.invoke(org.apache.cxf.message.Exchange, java.lang.Object, java.lang.reflect.Method, java.util.List) @bci=26, line=97 (Interpreted frame)
- org.apache.cxf.jaxws.AbstractJAXWSMethodInvoker.invoke(org.apache.cxf.message.Exchange, java.lang.Object, java.lang.reflect.Method, java.util.List) @bci=132, line=232 (Interpreted frame)
- org.apache.cxf.jaxws.JAXWSMethodInvoker.invoke(org.apache.cxf.message.Exchange, java.lang.Object, java.lang.reflect.Method, java.util.List) @bci=76, line=69 (Interpreted frame)
- org.apache.cxf.service.invoker.AbstractInvoker.invoke(org.apache.cxf.message.Exchange, java.lang.Object) @bci=215, line=75 (Interpreted frame)
- org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run() @bci=26, line=59 (Interpreted frame)
- java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=471 (Interpreted frame)
- java.util.concurrent.FutureTask.run() @bci=42, line=262 (Interpreted frame)
- org.apache.cxf.interceptor.ServiceInvokerInterceptor$2.run() @bci=46, line=126 (Interpreted frame)
- org.apache.cxf.workqueue.SynchronousExecutor.execute(java.lang.Runnable) @bci=1, line=37 (Interpreted frame)
- org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(org.apache.cxf.message.Message) @bci=165, line=131 (Interpreted frame)
- org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(org.apache.cxf.message.Message) @bci=137, line=307 (Compiled frame)
- org.apache.cxf.transport.ChainInitiationObserver.onMessage(org.apache.cxf.message.Message) @bci=416, line=121 (Interpreted frame)
- org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(javax.servlet.ServletConfig, javax.servlet.ServletContext, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=102, line=251 (Interpreted frame)
- org.apache.cxf.transport.servlet.ServletController.invokeDestination(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, org.apache.cxf.transport.http.AbstractHTTPDestination) @bci=55, line=234 (Interpreted frame)
- org.apache.cxf.transport.servlet.ServletController.invoke(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, boolean) @bci=319, line=208 (Interpreted frame)
- org.apache.cxf.transport.servlet.ServletController.invoke(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=4, line=160 (Interpreted frame)
- org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=42, line=171 (Interpreted frame)
- org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=161, line=293 (Interpreted frame)
- org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=3, line=212 (Interpreted frame)
- javax.servlet.http.HttpServlet.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=131, line=755 (Interpreted frame)
- org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=51, line=268 (Interpreted frame)
- org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=446, line=305 (Interpreted frame)
- org.apache.catalina.core.ApplicationFilterChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=101, line=210 (Interpreted frame)
- org.apache.catalina.core.StandardWrapperValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=783, line=222 (Interpreted frame)
- org.apache.catalina.core.StandardContextValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=166, line=123 (Interpreted frame)
- org.apache.catalina.authenticator.AuthenticatorBase.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=363, line=472 (Interpreted frame)
- org.apache.catalina.core.StandardHostValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=148, line=171 (Interpreted frame)
- org.apache.catalina.valves.ErrorReportValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=6, line=99 (Interpreted frame)
- org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=190, line=178 (Interpreted frame)
- org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response, org.wso2.carbon.tomcat.ext.valves.CompositeValve) @bci=3, line=47 (Interpreted frame)
- org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response, org.wso2.carbon.tomcat.ext.valves.CompositeValve) @bci=89, line=56 (Interpreted frame)
- org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response, org.wso2.carbon.tomcat.ext.valves.CompositeValve) @bci=26, line=47 (Interpreted frame)
- org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=345, line=141 (Interpreted frame)
- org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=102, line=156 (Interpreted frame)
- org.wso2.carbon.ui.valve.XSSValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=116, line=118 (Interpreted frame)
- org.wso2.carbon.ui.valve.CSRFValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=17, line=72 (Interpreted frame)
- org.apache.catalina.valves.AccessLogValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=6, line=936 (Interpreted frame)
- org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=11, line=52 (Interpreted frame)
- org.apache.catalina.core.StandardEngineValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=71, line=118 (Interpreted frame)
- org.apache.catalina.connector.CoyoteAdapter.service(org.apache.coyote.Request, org.apache.coyote.Response) @bci=196, line=407 (Interpreted frame)
- org.apache.coyote.http11.AbstractHttp11Processor.process(org.apache.tomcat.util.net.SocketWrapper) @bci=537, line=1004 (Compiled frame)
- org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(org.apache.tomcat.util.net.SocketWrapper, org.apache.tomcat.util.net.SocketStatus) @bci=158, line=589 (Interpreted frame)
- org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run() @bci=129, line=1653 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1145 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
Even though the HashSet is not thread safe, Many threads(around 100) getting blocked.And all threads have same stack trace in the thread dump. And the server is stuck. In that time, there are many requests coming into the server. For each request, One EntityManager is created.
EDIT The GenericDAO implementation is
Map<String, String> properties = new HashMap<String, String>();
properties.put("javax.persistence.jdbc.driver", "com.mysql.jdbc.Driver");
properties.put("javax.persistence.jdbc.url", "jdbc:mysql://"+dbProperties.getHost()+":"+dbProperties.getPort()+"/"+dbProperties.getDbName());
properties.put("javax.persistence.jdbc.user", dbProperties.getUser());
properties.put("javax.persistence.jdbc.password", dbProperties.getPassword());
entityManager = Persistence.createEntityManagerFactory("TestDAL", properties).createEntityManager();
