0

We have WebSphere environment with 2 Node Agents and 4 Application Servers. On high traffic one of the application servers stops responding to requests at it jumps to maximum web container thread counts.
On analyzing thread dump we found that approx 60% of threads are in runnable state with 20% each of Wait and Parked state.
We do not see any deadlock warning in thread dump. On closely look we have found that one of the web container thread owns the lock with below message:

Owns Monitor Lock on com/ibm/ws/classloader/ExtJarClassLoader@0x0A00000000FA6F30

Could someone help with the understanding of the above error and its resolution?

Shlok Thakur
  • 13
  • 1
  • 7
  • Do you see any Error in Websphere logs ? – Mr_Thorynque Jul 20 '16 at 07:45
  • could you provide more data like ffdc errors. – Mustafa DOGRU Jul 20 '16 at 08:31
  • In the logs we are seeing problem or scalability issue with Database Instance. With high volume traffic we are seeing database response time increasing lets say from 1sec to 5sec. The immediate interpretation was that we were choking at database which slowly and eventually taking app server to its max capacity. We doubled the resources at database end but the problem still persists. Now we see the above error in thread dump logs. – Shlok Thakur Jul 20 '16 at 08:49
  • @M.Dogru Could you please help us, how we can see ffdc errors? – Shlok Thakur Jul 20 '16 at 08:55
  • /AppServer/profiles//logs/ffdc – Mustafa DOGRU Jul 20 '16 at 09:07

1 Answers1

1

It is important to look at the stack trace of thread that owns the lock, and then look at the stack trace of all the other threads waiting on that lock. For ExtJarClassLoader, it almost certainly means that all the threads are attempting to perform loadClass operations. If many threads are attempting this and getting blocked, then this typically means the running code is attempting a failing loadClass operation, catching the ClassNotFoundException, and continuing. Creating and throwing ClassNotFoundException is expensive, so the code should typically be modified to cache the overall result (e.g., if it is trying a sequence of class loads, the positive/negative outcome should somehow be cached). This will of course be complicated if the code calling loadClass is a third party library.

Brett Kail
  • 33,593
  • 2
  • 85
  • 90
  • If more assistance is needed zip all the javacore files together into a single zip file and upload them to https://wait.ibm.com/ which will analyze the thread dumps and report where the application is bottlenecking. – Alexandre Polozoff Jul 26 '16 at 14:29