0

We recently upgraded our server from Tomcat 6 to Tomcat 9 and have been experiencing issues with the application going down every couple of hours. Some points I have gathered so far:

  • The system loads up without any errors and on the local machine and this only happens on production site.
  • The application works for 6-7 hours and then hits this white screen issue.
  • Once you restart the Tomcat server, the system is back again for a couple of hours and then the whole process is repeated.
  • It usually has connection issues after it's up for couple hours and it seems like the client connections in database seem to be really high when this happens.

This is what is logged in Tomcat. On debugging, the user is not null so I'm not exactly sure what the real issue is. I'm not sure if I need to redo the connections for the database or if it's a configuration issue with Tomcat.

26-Jan-2022 11:23:46.614 SEVERE [https-jsse-nio-443-exec-6] org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() for servlet [jsp] in context with path [] threw exception [An exception occurred processing [jsp/index.jsp] at line [86]

83: <%
84:     AccountInfo accountInfo = new AccountInfo();
85:     String user = testOBJECTS.getName();
86:     accountInfo.init(user);
87:   



Stacktrace:] with root cause
    java.lang.IllegalStateException: Failed to get connection..
        at test.library.TestConnectionFactory.getConnection(Unknown Source)
        at test.model.AccountInfo.init(Unknown Source)
        at org.apache.jsp.jsp.index_jsp._jspService(index_jsp.java:248)
        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:466)
        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:379)
        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:327)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1726)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Unknown Source)

Here is the code in index.jsp page where it's getting the user.

<jsp:useBean id="testOBJECTS" class="test.mainJspBean"/>
<jsp:useBean id="testTop" class="test.loggedInBean"/>
<jsp:useBean id="aircraftFolder" class="test.library.AircraftFolder"/>
<% 
  testOBJECTS.setLogin(request);
    if (testOBJECTS.getLoggedIn()) {
     testTop.setServletTop(request, testOBJECTS.getLoggedIn(), null, testOBJECTS.getName());

    AccountInfo accountInfo = new AccountInfo();
    String user = testOBJECTS.getName();
    accountInfo.init(user);
    
    }
%>

And here is the MainJspBean.java class where it's setting the user.

package test;

import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;

import javax.servlet.http.Cookie;
import javax.servlet.http.HttpServletRequest;

import org.apache.logging.log4j.Logger; 
import org.apache.logging.log4j.LogManager;

import test.library.SubscribeConnectionFactory;
import test.loginLibraries.Login;

public class mainJspBean
{
    static Logger log = LogManager.getLogger(mainJspBean.class.getName());
    private String clientPage = "";
    private boolean loggedIn = false;
    private String name = "";

    public String getLogin()
    {
        return clientPage;
    }

    public void setLogin(HttpServletRequest request) throws SQLException {
        String user = (String) request.getSession().getAttribute(Login.UID);
        String pass = (String) request.getSession().getAttribute(Login.PASS);
        boolean loggedin = loggedInBean.isLoggedIn(user, pass);
        if (loggedin) {
            this.loggedIn = loggedin;
            name = user;
            System.out.println("name: " + name);
            
            clientPage += "<div align=\"justify\" class=\"style1\"><a href=\"https://" + IPs.WebAddress() + "/jsp/index.jsp\">TEST Online</a></div>\n";
        } else {
            clientPage += "<br><FORM  autocomplete=\"off\" METHOD=\"post\" ACTION=\"https://"+ IPs.WebAddress() +"/test/login\" onSubmit=\"return checkWhiteSpace()\">\n" +
                      "<table border=\"0\" cellpadding=\"0\" cellspacing=\"0\" align=\"right\" valign=\"top\">\n" +
                      "<tr>\n" +
                      "<td nowrap align=\"right\" ><span class=\"topText\" align=\"right\">User Name:&nbsp;</span><input type=\"text\" name=\"uid\" id=\"uid\" size=\"8\" ></td>\n" +
                      "</tr>\n" +
                      "<tr>\n" +
                      "<td nowrap align=\"right\" ><span class=\"topText\" align=\"right\">Password:&nbsp;</span><input type=\"password\" name=\"auth\" size=\"8\"></td>\n" +
                      "</tr>\n" +
                      "<tr>\n" +
                      "<td nowrap colspan=\"2\" align=\"right\" ><input type=\"checkbox\" name=\"remember\" value=\"1\" checked>&nbsp;<span " + "class=\"topText\">Remember me.</span></td>\n" +
                      "</tr>\n" +   
                      "<tr>\n" +
                      "<td align=\"right\" colspan=\"2\"><input type=\"submit\" class=\"IP\" name=\"login\" value=\"Login\">\n" +                           
                      "<a href=\"https://" + IPs.WebAddress() + "/payAsYouGoSignUp.jsp\">Register</a></td>\n" +
                      "</tr>\n" +
                
                      "</table>\n" +
                      "<script type=\"text/javascript\">\n" +
                      "<!--\n" +
                      "function checkWhiteSpace() {\n" +
                      "  var s = document.getElementById(\"uid\").value;\n"+
                      "   // Check for white space\n" +
                      "  if (s.indexOf(\" \") > -1) {\n" +
                      "      jAlert(\"The User Name that you entered contains a white space.\");\n" +
                      "      return false;\n" +
                      "  }\n" +
                      "  return true;\n" +
                      "}\n" +
                      "//-->\n" +
                      "</script>\n" +
                      "</form>\n";          
        }
      }
    

    public String getWebAddress()
    {
        return IPs.WebAddress();
    }

    public boolean getLoggedIn()
    {
        return loggedIn;
    }

    public String getName()
    {
        return name;
    }
}

And here is the actual TestConnection page.

public class TestConnectionFactory {
    static final Runtime RUNTIME = Runtime.getRuntime();
    static Logger log = LogManager.getLogger(TestConnectionFactory.class.getName());
    DataSource ds = null;
    long connectCount;
    /**
     * A handle to the unique Singleton instance.
     */
    static private TestConnectionFactory instance = null;

    /**
     * The constructor could be made private
     * to prevent others from instantiating this class.
     * But this would also make it impossible to
     * create instances of Singleton subclasses.
     */
    protected TestConnectionFactory() {
        log.debug("<init> freeMemory:"+RUNTIME.freeMemory());
        if (ds == null) {
            ds = getDataSource();
        }
    }

    /**
     * @return The unique instance of this class.
     */
    static public synchronized TestConnectionFactory getInstance() {
        if (instance == null) {
            log.info("getInstance():creating new instance");
            instance = new TestConnectionFactory();
            log.debug("getInstance():reusing existing instance");
        }
        return instance;
    }

    public Connection getConnection() {
        Connection connection = null;
        PreparedStatement pstmt = null;
        ResultSet rs = null;

            try {
                connection = ds.getConnection();

            } catch (SQLException e) {
                e.printStackTrace();
            }
            return connection; 
           
    }

Something I tried was referring to this Java web app in tomcat periodically freezes up post as the issues sounded similar to my application but the fix didn't work. Things I tried:

  • Changing the maxWaitMillis and maxActive in the context.xml file.
  • Increasing total number of client database connections.
  • Set traps for null user in the Java code.
ace23
  • 142
  • 1
  • 16
  • 1
    At first view, when this happens, you can connect to the database (server) from the machine (shell) where tomcat is running ? The database is up ? What OS ? `index.jsp` is the only endpoint ? – PeterMmm Jan 26 '22 at 18:31
  • @PeterMmm I think so. I only have to restart the tomcat and not mysql so I am assuming it is but I need to double check again next time it happens. It's on Windows. And it's a pooled connection so it's being used in quite a few places. This is one of the many pages that seems to throw this error message. – ace23 Jan 26 '22 at 18:48
  • Do you really close the connection everywhere `TestConnectionFactory.getConnection()` is used ? – PeterMmm Jan 26 '22 at 19:21
  • My first step would be to stop using connection pooling. Use fresh connections to see if all goes well. – Basil Bourque Jan 26 '22 at 22:12
  • 1
    Your getConnection method has very brittle resource management, and it leaks connections if the validation fails. Also most connection pooling data sources will provide these checks for you: no need to roll your own. My guess is something changed with the data source you use (e.g. it no longer reclaims leaked connections, or something like that). – Mark Rotteveel Jan 27 '22 at 14:26
  • @MarkRotteveel Rewrote the connection page. That seems to fix my issue. Thank you! – ace23 Feb 08 '22 at 16:52

1 Answers1

0

Updated the getConnection() method seems to fix my issue.

public Connection getConnection() {
    Connection connection = null;
    PreparedStatement pstmt = null;
    ResultSet rs = null;

        try {
            connection = ds.getConnection();

        } catch (SQLException e) {
            e.printStackTrace();
        }
        return connection; 
       
}
ace23
  • 142
  • 1
  • 16