1

I am working on a java based web app that uses Tomcat 6 as the web server. Normally, we have a MySQL backend, but for a few instances we need to use MS SQL to serve our clients. When performing our latest upgrade to one of these MS SQL instances, I began getting java out of memory errors when accessing certain pages of the web app. Usually occurring when returning a larger result set, but not always.

I do not receive this error on the same web app set-up as a local instance on my PC, which leads me to believe that this is an environment issue. I have tried numerous different Java settings in the Tomcat 6 configuration without any luck fixing the issue.

The environment is a 32-bit machine with 4GB of RAM. I have configured Tomcat 6 with the following: -XX:MaxPermSize=512m -Xnoclassgc Initial memory pool: 1024 Maximum memory pool: 1024

I have also tried increasing the MaxPermSize, but this causes Tomcat to not be able to start.

As well, I have tried setting -Xms -Xmx, to 256m each, 512m each, etc.

My local machine is 64-bit with 4GB RAM. I have configured Tomcat 6 with the following: -XX:MaxPermSize=512m -Xnoclassgc Initial memory pool: 1024 Maximum memory pool: 1024

Could this be an issue with an SQL statement in the new upgrade? I don't don't believe this is the case because the error does not happen on my local instance. I believe it's environment related, but can't figure out the issue.

Does anyone have any advice?

Here is the logged error: 

2012-09-19 11:44:54,745 [ http-80-7] ERROR org.apache.jsp.siteManager.util.error_jsp: javax.servlet.ServletException: java.lang.OutOfMemoryError: unable to create new native thread
at org.apache.jasper.runtime.PageContextImpl.doHandlePageException(PageContextImpl.java:862)
at org.apache.jasper.runtime.PageContextImpl.handlePageException(PageContextImpl.java:791)
at org.apache.jsp.doc.doc.index_jsp._jspService(index_jsp.java:1106)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:436)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:374)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:302)
at com.djinnsoft.jade.sitetree.Sitetree.forward(Sitetree.java:405)
at com.djinnsoft.jade.sitetree.Sitetree.forward(Sitetree.java:344)
at com.djinnsoft.jade.sitetree.RootServlet.service(RootServlet.java:124)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.springboard.ram.report.ReportSecurityFilter.doFilter(ReportSecurityFilter.java:70)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:861)
at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:579)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1584)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Unknown Source)
at com.microsoft.sqlserver.jdbc.TimeoutTimer.start(Unknown Source)
at com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(Unknown Source)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteStatement(Unknown Source)
at com.microsoft.sqlserver.jdbc.SQLServerStatement$StmtExecCmd.doExecute(Unknown Source)
at com.microsoft.sqlserver.jdbc.TDSCommand.execute(Unknown Source)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(Unknown Source)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(Unknown Source)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(Unknown Source)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeQuery(Unknown Source)
at org.apache.tomcat.dbcp.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)
at org.apache.tomcat.dbcp.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)
at com.djinnsoft.jade.data.record.SQLUtil.ExecuteQuery(SQLUtil.java:236)
at com.djinnsoft.jade.data.record.SQLUtil.ExecuteQuery(SQLUtil.java:207)
at com.djinnsoft.jade.data.record.RecordSource.find(RecordSource.java:184)
at com.djinnsoft.jade.data.record.RecordSource.find(RecordSource.java:148)
at com.djinnsoft.jade.content.data.DatabaseContentSource.createVersions(DatabaseContentSource.java:804)
at com.djinnsoft.jade.content.data.DatabaseContentSource.access$2900(DatabaseContentSource.java:42)
at com.djinnsoft.jade.content.data.DatabaseContentSource$8.process(DatabaseContentSource.java:747)
at com.djinnsoft.jade.data.record.SQLEnvironment.execute(SQLEnvironment.java:70)
at com.djinnsoft.jade.content.data.DatabaseContentSource.findImpl(DatabaseContentSource.java:672)
at com.djinnsoft.jade.content.GenericContentSource.findImpl(GenericContentSource.java:1077)
at com.djinnsoft.jade.content.GenericContentSource.find(GenericContentSource.java:1186)
at com.djinnsoft.jade.content.GenericContentSource.find(GenericContentSource.java:1200)
at com.djinnsoft.jade.content.GenericContentSource.find(GenericContentSource.java:1025)
at com.djinnsoft.jade.content.taglib.ContentBaseTag.findItem(ContentBaseTag.java:166)
at com.djinnsoft.jade.content.taglib.ContentTag.doStartTag(ContentTag.java:500)
at org.apache.jsp.doc.doc.index_jsp._jspx_meth_content_005faction_005f12(index_jsp.java:7637)
at org.apache.jsp.doc.doc.index_jsp._jspx_meth_c_005fforEach_005f6(index_jsp.java:6299)
at org.apache.jsp.doc.doc.index_jsp._jspx_meth_c_005fotherwise_005f3(index_jsp.java:4816)
at org.apache.jsp.doc.doc.index_jsp._jspx_meth_c_005fchoose_005f3(index_jsp.java:4395)
at org.apache.jsp.doc.doc.index_jsp._jspx_meth_c_005fotherwise_005f2(index_jsp.java:4363)
at org.apache.jsp.doc.doc.index_jsp._jspx_meth_c_005fchoose_005f2(index_jsp.java:4278)
at org.apache.jsp.doc.doc.index_jsp._jspx_meth_forms_005fpopulateResult_005f0(index_jsp.java:4244)
at org.apache.jsp.doc.doc.index_jsp._jspService(index_jsp.java:961)
... 32 more
steve.westfall
  • 159
  • 1
  • 12
  • 2
    Did you see this? http://stackoverflow.com/questions/1427615/sql-server-jdbc-unable-to-create-new-native-thread – Martin Wilson Sep 19 '12 at 16:24
  • If after reading what Martin linked you still don't have luck, try these options "-Xms512M -Xmx1024M -XX:MaxPermSize=256M" – Sully Sep 19 '12 at 16:26
  • sometimes it isn't enough the -Xmx1024M for me, and I need 2G :( on a big, nice, customized reports and internally stored objects needed for report. –  Sep 19 '12 at 16:29
  • OK, tried setting -Xss128M like in the link Martin suggested, but it didn't work. Then I tried the options D3mon-1stVFW suggested, with no luck either. Now, maybe it's just me. I am setting these by using Configure Tomcat. Is this correct? Or should I be setting them as runtime parameters through the Java Control panel. Does where I set them matter? – steve.westfall Sep 19 '12 at 17:15
  • Sorry, that should have read -Xss128k above – steve.westfall Sep 19 '12 at 17:25

2 Answers2

0

2012-09-19 11:44:54,745 [ http-80-7] ERROR org.apache.jsp.siteManager.util.error_jsp: javax.servlet.ServletException: java.lang.OutOfMemoryError: unable to create new native thread

This usually indicates that you got the OOM in your native memory and not the java heap. It means that the OS was not able to allocate the thread.
This could either mean that:

  • You have too many threads running and you exceeded some OS limit (you are using a 64-bit machine which has different thread limits than a 32-bit machine) or

  • One of your threads used too much memory on initialization and crashed (I can not really describe this more acuretly).

You say that you are not able to reproduce this but could this be, that it is the load of the web server that you don't have in order to be able to reproduce this locally? I mean perhaps in the deployed server instance the load was too heavy, too many concurrent requests (too many application threads?) and this caused the crash?

Cratylus
  • 52,998
  • 69
  • 209
  • 339
  • How can I confirm this? Will it be logged in the server event logs? Catalina logs? Any advice is appreciated. – steve.westfall Sep 19 '12 at 17:49
  • This is hard to tell.From your application logs and possibly the catalina logs you should be able to figure out, if your server is under a heavy load.If it is you could try to do a stress run of your application locally in a **32-bit** system and **not** a 64-bit as you are currently doing.Additionally, the OOM also produces a heap dump. Have you inspected that dump file with a tool? – Cratylus Sep 19 '12 at 18:09
  • Thanks for the advice Cratylus, it is appreciated, but I already found what was causing the issue. – steve.westfall Sep 19 '12 at 18:15
0

OK, so, I found the solution to the problem. In our new version of the web app, there is a context parameter that limits the display size of the results sets and paginates them at that limit. As I said earlier, this issue only seemed to be happening on larger result sets. The default setting for MySQL backends is 500, but apparently the MS SQL backend needs this to be set much lower. Setting it to 20 eliminated the out of memory errors. Thanks to all who offered advice. It was greatly appreciated.

steve.westfall
  • 159
  • 1
  • 12