I have a web app running on bluemix liberty. If offers a rest api that retrieves information from a db2 instance via jdbc. I'm observing that after about 7 days all my rest api queries are failing. Looking at logs, I am seeing NoClassDef exception thrown by DriverManager.getConnection method. There is also an associated FileNotFound exception (with reason 'Too many open files') for an ffdc exception summary file, preventing it from being written. Excerpt from messages.log included at the bottom.
Looking at my code, I saw that I'm not closing a jdbc connection in a rest api request that was being invoked every 5 minutes for the last 7 days. Could this lead to exceeding the limit on open file descriptors for the wlp server (the limit is about 16000)?
I tried the following steps to verify this:
- Set property 'BLUEMIX_APP_MGMT_ENABLE' to 'devconsole + shell'. (cf set-env BLUEMIX_APP_MGMT_ENABLE devconsole+shell).
- Restaged app.
- Opened shell to app env via 'https:///bluemix-debug/shell'.
- Found out pid of wlp server. Ran 'lsof -p pid | wc -l' to get count of open files.
- Ran the rest api request with unclosed connection 500 times.
- Took a count of open files for above pid again.
I observed that there was an increase in open files count, but temporary. The number went down gradually (by the time all 500 requests had completed) to almost the same as before. I was expecting the spike in open files count to last longer.
Is this approach correct (i.e. leak is happening elsewhere)? I had tried with jconsole too but it was not showing open file descriptor count.
[2/9/16 3:00:11:773 GMT] 0000fd52 com.ibm.ws.webcontainer.util.ApplicationErrorUtils E SRVE0777E: Exception thrown by application class 'org..apache.wink.server.internal.RequestProcessor.handleRequest:195'
javax.servlet.ServletException: java.lang.NoClassDefFoundError: com/ibm/db2/jcc/am/uo
at org.apache.wink.server.internal.RequestProcessor.handleRequest(RequestProcessor.java:195)
at com.ibm.websphere.jaxrs.server.IBMRestServlet..service(IBMRestServlet.java:107)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1287)
at com.ibm.ws.webcontainer.servlet..ServletWrapper.handleRequest(ServletWrapper.java:778)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:475)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1161)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:81)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:928)
at com.ibm.ws.webcontainer.osgi.DynamicVirtualHost$2.run(DynamicVirtualHost.java:262)
at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink$TaskWrapper.run(HttpDispatcherLink.java:955)
at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.ready(HttpDispatcherLink.java:341)
at com.ibm..ws.http.channel.internal.inbound.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:470)
at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleNewRequest(HttpInboundLink.java:404)
at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.processRequest(HttpInboundLink.java:284)
at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.ready(HttpInboundLink.java:255)
at com.ibm.ws.tcpchannel.internal.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback..java:174)
at com.ibm.ws.tcpchannel.internal.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:83)
at com.ibm.ws.tcpchannel.internal.WorkQueueManager.requestComplete(WorkQueueManager.java:504)
at com.ibm.ws.tcpchannel.internal.WorkQueueManager.attemptIO(WorkQueueManager.java:574)
at com.ibm.ws.tcpchannel.internal.WorkQueueManager.workerRun(WorkQueueManager.java:929)
at com.ibm.ws.tcpchannel.internal.WorkQueueManager$Worker.run(WorkQueueManager.java:1018)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NoClassDefFoundError: com/ibm/db2/jcc/am/uo
at com.ibm.db2.jcc.t4.xb.a(xb.java:433)
at com.ibm.db2.jcc.t4.xb.<init>(xb.java:90)
at com.ibm.db2.jcc.t4.a.z(a.java:347)
at com.ibm.db2.jcc.t4.b.a(b.java:1974)
at com.ibm.db2.jcc.am.ib.a(ib.java:691)
at com.ibm.db2.jcc.am.ib.<init>(ib.java:644)
at com.ibm.db2.jcc.t4.b..<init>(b.java:330)
at com.ibm.db2.jcc.DB2SimpleDataSource.getConnection(DB2SimpleDataSource.java:231)
at com.ibm.db2.jcc.DB2SimpleDataSource.getConnection(DB2SimpleDataSource.java:197)
at com.ibm.db2.jcc.DB2Driver.connect(DB2Driver.java:472)
at com.ibm.db2.jcc.DB2Driver.connect(DB2Driver.java:113)
at java.sql.DriverManager.getConnection(Unknown Source)
at java.sql.DriverManager.getConnection(Unknown Source)
at com.ibm.biginsights.bluemix.infra.util.RepositoryHelper.getConnection(RepositoryHelper.java:58)
at com.ibm.biginsights.bluemix.infra.dao.JdbcDAOFactoryImpl.initConnection(JdbcDAOFactoryImpl.java:39)
at com.ibm.biginsights.bluemix.infra.pojo.MonitoringAPI.getStatus(MonitoringAPI.java:48)
at com.ibm.biginsights.bluemix.infra.rest.MonitoringService.getStats(MonitoringService.java:53)
at sun.reflect.GeneratedMethodAccessor82.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.wink.server.internal.handlers.InvokeMethodHandler.handleRequest(InvokeMethodHandler.java:63)
at org.apache.wink.server.handlers.AbstractHandler.handleRequest(AbstractHandler.java:33)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
at org.apache.wink.server.internal.handlers.CreateInvocationParametersHandler.handleRequest(CreateInvocationParametersHandler.java:54)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
at org.apache.wink.server.handlers.AbstractHandler.handleRequest(AbstractHandler.java:34)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
at org.apache.wink.server.internal.handlers.FindResourceMethodHandler.handleResourceMethod(FindResourceMethodHandler.java:151)
at org.apache.wink.server.internal.handlers.FindResourceMethodHandler.handleRequest(FindResourceMethodHandler.java:65)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
at org.apache.wink.server.internal.handlers.FindRootResourceHandler.handleRequest(FindRootResourceHandler.java:95)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
at org.apache.wink..server.internal.handlers.HeadMethodHandler.handleRequest(HeadMethodHandler..java:53)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
at org.apache.wink.server.internal.handlers.OptionsMethodWADLHandler.handleRequest(OptionsMethodWADLHandler.java:51)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
at org.apache.wink.server.internal.handlers.SearchResultHandler.handleRequest(SearchResultHandler.java:33)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
at org.apache.wink.server.internal.log.ResourceInvocation.handleRequest(ResourceInvocation.java:92)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
at org.apache.wink.server.internal.log.Requests.handleRequest(Requests.java:76)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
at org.apache.wink.server.handlers.AbstractHandlersChain.run(AbstractHandlersChain.java:60)
at org.apache.wink.server.internal.RequestProcessor.handleRequestWithoutFaultBarrier(RequestProcessor.java:207)
at org..apache.wink.server.internal.RequestProcessor.handleRequest(RequestProcessor.java:154)
... 24 more
[2/9/16 3:00:11:774 GMT] 0000fd52 SystemErr R java.io.FileNotFoundException: ../../../../../logs/ffdc/exception_summary_16.02.05_08.25.38.0.log (Too many open files)
[2/9/16 3:00:11:774 GMT] 0000fd52 SystemErr R at java.io.FileOutputStream.open0(Native Method)
[2/9/16 3:00:11:775 GMT] 0000fd52 SystemErr R at java.io.FileOutputStream.open(Unknown Source)
[2/9/16 3:00:11:775 GMT] 0000fd52 SystemErr R at java.io.FileOutputStream.<init>(Unknown Source)
[2/9/16 3:00:11:775 GMT] 0000fd52 SystemErr R at java.io.FileOutputStream.<init>(Unknown Source)
[2/9/16 3:00:11:775 GMT] 0000fd52 SystemErr R at com.ibm.ws.kernel..boot.logging.DefaultFileStreamFactory.createOutputStream(DefaultFileStreamFactory.java:28)
[2/9/16 3:00:11:775 GMT] 0000fd52 SystemErr R at com.ibm.ws.kernel.boot.logging..TextFileOutputStreamFactory.createOutputStream(TextFileOutputStreamFactory..java:66)
[2/9/16 3:00:11:775 GMT] 0000fd52 SystemErr R at com.ibm.ws.kernel.launch.internal.LauncherDelegateImpl$1.createOutputStream(LauncherDelegateImpl.java:59)
[2/9/16 3:00:11:776 GMT] 0000fd52 SystemErr R at com.ibm.ws.logging.internal.impl.BaseFFDCService.createSummaryStream(BaseFFDCService.java:430)
[2/9/16 3:00:11:776 GMT] 0000fd52 SystemErr R at com.ibm.ws.logging.internal.impl.BaseFFDCService.logSummary(BaseFFDCService.java:300)
[2/9/16 3:00:11:776 GMT] 0000fd52 SystemErr R at com.ibm.ws.logging.internal.impl.BaseFFDCService.logSummary(BaseFFDCService.java:274)
[2/9/16 3:00:11:776 GMT] 0000fd52 SystemErr R at com.ibm.ws.logging.internal.impl.BaseFFDCService.log(BaseFFDCService.java:237)
[2/9/16 3:00:11:776 GMT] 0000fd52 SystemErr R at com.ibm.ws.logging.internal.impl.BaseFFDCService.processException(BaseFFDCService.java:160)
[2/9/16 3:00:11:776 GMT] 0000fd52 SystemErr R at com.ibm.ws.ffdc.FFDCFilter.processException(FFDCFilter.java:45)
[2/9/16 3:00:11:777 GMT] 0000fd52 SystemErr R at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1226)
[2/9/16 3:00:11:777 GMT] 0000fd52 SystemErr R at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:81)
[2/9/16 3:00:11:777 GMT] 0000fd52 SystemErr R at com.ibm.ws.webcontainer.WebContainer..handleRequest(WebContainer.java:928)
[2/9/16 3:00:11:777 GMT] 0000fd52 SystemErr R at com.ibm.ws.webcontainer.osgi.DynamicVirtualHost$2.run(DynamicVirtualHost.java:262)
[2/9/16 3:00:11:777 GMT] 0000fd52 SystemErr R at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink$TaskWrapper.run(HttpDispatcherLink.java:955)
[2/9/16 3:00:11:777 GMT] 0000fd52 SystemErr R at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.ready(HttpDispatcherLink.java:341)
[2/9/16 3:00:11:777 GMT] 0000fd52 SystemErr R at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:470)
[2/9/16 3:00:11:778 GMT] 0000fd52 SystemErr R at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleNewRequest(HttpInboundLink.java:404)
[2/9/16 3:00:11:778 GMT] 0000fd52 SystemErr R at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.processRequest(HttpInboundLink.java:284)
[2/9/16 3:00:11:778 GMT] 0000fd52 SystemErr R at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.ready(HttpInboundLink.java:255)
[2/9/16 3:00:11:778 GMT] 0000fd52 SystemErr R at com.ibm.ws.tcpchannel.internal.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:174)
[2/9/16 3:00:11:778 GMT] 0000fd52 SystemErr R at com.ibm.ws.tcpchannel.internal.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:83)
[2/9/16 3:00:11:778 GMT] 0000fd52 SystemErr R at com.ibm.ws.tcpchannel.internal.WorkQueueManager.requestComplete(WorkQueueManager.java:504)
[2/9/16 3:00:11:779 GMT] 0000fd52 SystemErr R at com.ibm.ws.tcpchannel.internal.WorkQueueManager.attemptIO(WorkQueueManager.java:574)
[2/9/16 3:00:11:779 GMT] 0000fd52 SystemErr R at com.ibm.ws.tcpchannel.internal.WorkQueueManager.workerRun(WorkQueueManager.java:929)
[2/9/16 3:00:11:779 GMT] 0000fd52 SystemErr R at com.ibm.ws.tcpchannel.internal.WorkQueueManager$Worker.run(WorkQueueManager.java:1018)
[2/9/16 3:00:11:779 GMT] 0000fd52 SystemErr R at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[2/9/16 3:00:11:779 GMT] 0000fd52 SystemErr R at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[2/9/16 3:00:11:779 GMT] 0000fd52 SystemErr R at java.lang.Thread.run(Unknown Source)
