JRockit getting stuck

Question

We've been having issues with the JRockit jvm getting stuck with this stack:

"[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" id=45 idx=0x94 tid=9944 prio=1 alive, in native, daemon
    at jrockit/vm/Allocator.nativeGetNewTLA()V(Native Method)
    at jrockit/vm/Allocator.getNewTLA(Allocator.java:788)[optimized]
    at jrockit/vm/Allocator.allocLargerThanFreeTLA(Allocator.java:816)[inlined]
    at jrockit/vm/Allocator.allocSlowCaseInner(Allocator.java:930)[inlined]
    at jrockit/vm/Allocator.allocSlowCase(Allocator.java:776)[optimized]
    at oracle/jdbc/driver/T4CMAREngine.unmarshalCLRforREFS(T4CMAREngine.java:2024)[optimized]
    at oracle/jdbc/driver/T4CTTIoer.unmarshal(T4CTTIoer.java:160)[optimized]
    at oracle/jdbc/driver/T4C8Oall.receive(T4C8Oall.java:727)[optimized]
    at oracle/jdbc/driver/T4CPreparedStatement.doOall8(T4CPreparedStatement.java:216)[inlined]
    at oracle/jdbc/driver/T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:955)[optimized]
    at oracle/jdbc/driver/OracleStatement.executeMaybeDescribe(OracleStatement.java:1060)[optimized]
    at oracle/jdbc/driver/T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:839)[optimized]
    at oracle/jdbc/driver/OracleStatement.doExecuteWithTimeout(OracleStatement.java:1132)[optimized]
    at oracle/jdbc/driver/OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3316)[optimized]
    at oracle/jdbc/driver/OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3361)[optimized]
    ^-- Holding lock: oracle/jdbc/driver/T4CPreparedStatement@0x1d8f8268[thin lock]
    ^-- Holding lock: oracle/jdbc/driver/T4CConnection@0x14d68fd8[thin lock]
    at weblogic/jdbc/wrapper/PreparedStatement.executeQuery(PreparedStatement.java:97)[optimized]

It seems to be stuck trying to allocate memory. According to our monitoring tools, the heap usage was around 14% and less than 20% on the time before the server got stuck.

This is the java version:

java version "1.5.0_14"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_14-b03)
BEA JRockit(R) (build R27.5.0-110_o-99226-1.5.0_14-20080528-1505-linux-x86_64, compiled mode)

These are the JVM settings we're using

-Xms256m -Xmx3072m -Xverify:none

Any ideas what could be causing this issue?

Take a hprof heap dump and run it through the Eclipse Memory analyzer Tool - this could be a memory leak due to some db connection pool operations not reeleasing the resource — JoseK, Jan 28 '13 at 05:14
When the JVM is stuck, I can't even take a heap dump, just a thread dump, unfortunately. — Andres Olarte, Feb 16 '13 at 19:48

score 3 · Accepted Answer · answered Mar 12 '13 at 13:57

It looks like it is stuck when trying to allocate additional TLA (thread local area) space. The thread local area (TLA) is a chunk of free space reserved on the heap or the nursery and given to a thread for its exclusive use. A thread can allocate small objects in its own TLA without synchronizing with other threads. When the TLA gets full the thread simply requests a new TLA. The thread get stuck because jrockit fails to allocate memory.

From the stacktrace this happens when you are reading data from the database, so it probably means that there was not enough TLA space for the read objects. Is this query reading a lot of records from the database?

You can try solving this issue by tuning the TLA size

score 1 · Answer 2 · answered Jan 27 '13 at 16:33

1

Most probably the problem is related to a DB operation: Holding lock: oracle/jdbc/driver/T4CPreparedStatement@0x1d8f8268[thin lock].

Have you checked whether a DB operation was under execution at the time your server experienced the mentioned issue?

You may check the following:

Long running Query
Problems related to the DB server causing a DB to respond slowly
DB operation which tries to load large chunks of data from a DB

answered Jan 27 '13 at 16:33

Apostolos Emmanouilidis

7,179
1
24
35

Wouldn't this lock only prevent access to the database? Right now when the problem happens, this is the only thread that's stuck in the JDBC code, no other thread is trying to access the database. And even connections that don't need the database (for example a connection to the weblogic console) get completely stuck. – Andres Olarte Jan 29 '13 at 14:34

JRockit getting stuck

2 Answers2