Websphere Server threads getting hung

Question

We have an application where we use an H2 embedded database to store the data. We have a synchronized write method which does DB inserts. Since the H2 DB is a small Java embedded DB, we use "synchronized" on the write method to handle the transaction management in embedded DB rather than in DB.

But during heavy load, we could see that the write thread is getting hung. We are not sure for which resource, this thread is getting hung.

Please look at this snippet of code:

public synchronized int write(IEvent event)  {
  String methodName = "write";
  Connection conn  = null;
  PreparedStatement updtStmt = null;
  Statement stmt = null;
  ResultSet rSet = null;
  int status = 0;
  try {
    dbConnect.checkDBSizeExceed();
    conn = dbConnect.getConnection();
    updtStmt =  conn.prepareStatement(insertQuery);
    updtStmt.setString(1, (String) event.getAttributeValue());
       ......
    updtStmt.setString(30, (String) event.getAttributeValue());
    updtStmt.setBoolean(31, false);
    status = updtStmt.executeUpdate();
  }catch(SQLException ex){
    logger.log(methodName,logger.print(ex),Logger.ERROR);
  } catch(Exception ex){
    logger.log(methodName,logger.print(ex),Logger.ERROR);
  } finally {
    try {
      if (updtStmt != null)
        updtStmt.close();
      if (conn != null)
        conn.close();
    }catch(SQLException ex) {
      logger.log(methodName,logger.print(ex),Logger.ERROR);
      return status;
    }
    return status;
  }
}

We have multiple write methods which can access this DB. From the code we could see that the code is straightforward. But we are not sure where the resource is locked.

Another problem is in the thread dump in the (Websphere) system.out, we could see the thread stacktrace as below.

[6/15/12 3:13:38:225 EDT] 00000032 ThreadMonitor W   WSVR0605W: Thread "WebContainer : 3" (00000066) has been active for 632062 milliseconds and may be hung.  There is/are 2 
thread(s) in total in the server that may be hung.
    at com.xxxx.eaws.di.agent.handlers.AuditEmbeddedDBHandler.store(Unknown Source)
    at com.xxxx.eaws.di.agent.eventlogger.2LoggerImpl.logEvent(Unknown Source)
    at com.xxxx.eecs.eventlogger.EventLoggerAdapter.logAuditEvent(EventLoggerAdapter.java:682)
    at com.xxxx.eecs.eventlogger.EventLoggerAdapter.logEvent(EventLoggerAdapter.java:320)
    at com.xxxx.eecs.eventlogger.EventLogger.logEventInternal(EventLogger.java:330)
    at com.xxxx.eecs.eventlogger.EventLogger.logEvent(EventLogger.java:283)
    at com.ibm.wps.auth.impl.ImplicitLoginFilterChainImpl.login(ImplicitLoginFilterChainImpl.java:55)
    at com.ibm.wps.auth.impl.AuthenticationFilterChainHandlerImpl.invokeImplicitLoginFilterChain(AuthenticationFilterChainHandlerImpl.java:393)
    at com.ibm.wps.auth.impl.InitialAuthenticationHandlerImpl.checkAuthentication(InitialAuthenticationHandlerImpl.java:204)
    at com.ibm.wps.state.phases.PhaseManagerImpl.callInitialAuthenticationHandler(PhaseManagerImpl.java:240)

In the above stacktrace, I need to know the reason why I am getting "Unknown Source" in the stack trace. Those jars are available in the class path and we also have the H2.jar in the classpath. We are not sure why, if the thread gets in hung in H2, we are not able to get the thread stacktrace.

If not, I also need to know why the thread stack trace is showing "Unknown Source".

Appreciate your help.

Thanks in advance.

The stack trace you're showing is not in the write() method you're showing. You also might try forcing a javacore to see where all your threads are sitting. — dbreaux, Oct 16 '15 at 14:28

score 0 · Answer 1 · edited Jul 01 '12 at 16:56

0

Are you using ejbs? How do you get the connection? Is it injected by the aop server? Do you retrieve it from jndi? You should Not synchronize the method. Even if it is an embedded db you should rely on the app server facilities. You need to configure the connection as a datasource, even if your db is in memory. If you want a serialized write on the db you need to configure the connection pool to the serialized ansi isolation level (there are 4 ansi isolation levels). In this way you should obtain the same effect in a managed environment (the app server) without the synchronized, that should be avoided inside an app server.

edited Jul 01 '12 at 16:56

Thomas Mueller

48,905
14
116
132

answered Jul 01 '12 at 14:13

JayZee

851
1
10
17

This is embedded DB is not in remote location it is physically in the same server. IT is a kind of temporary location to store data locally. I use the Datasource connections given by H2. the data retrieval is via service timers which will run every 30 secs and retrieve the data and sent it to remote server for further processing via EJB protocol. – Sam Jul 02 '12 at 06:27

score 0 · Answer 2 · answered Jul 02 '12 at 02:57

0

Unknown Source typically implies that the line numbers are available.

When you compile, the compiler can add in debug information like line numbers. If they are not present in the JAR or .class files, the JVM can't provide you that information.

HTH

answered Jul 02 '12 at 02:57

Manglu

10,744
12
44
57

score 0 · Answer 3 · answered Feb 19 '14 at 10:33

It looks like the conn = dbConnect.getConnection(); is waiting for more than 60000ms.

The error thrown by WAS is because resource adapter has one mechanism poll period. It is considered the rate (in milliseconds) at which to poll the enterprise information system (EIS) event store for new inbound events. The poll cycle is established at a fixed rate, meaning that if execution of the poll cycle is delayed for any reason, the next cycle will occur immediately to “catch up”. During the poll period, the polling thread will be sleeping. Once the time is calculated to 60000 milliseconds, the WebSphere Application Server thread monitor regards this polling thread as hung and throws the exception.

Websphere Server threads getting hung

3 Answers3