13

I have Java EE based application running on tomcat and I am seeing that all of a sudden the application hangs after running for couple of hours.

I collected the thread dump from the application just before it hangs and put it in TDA for analysis:

enter image description here

TDA (Thread Dump Analyzer) gives the following message for the above monitor:

A lot of threads are waiting for this monitor to become available again.
This might indicate a congestion. You also should analyze other locks 
blocked by threads waiting for this monitor as there might be much more 
threads waiting for it.

And here is the stacktrace of the thread highlighted above:

"MY_THREAD" prio=10 tid=0x00007f97f1918800 nid=0x776a 
             waiting for monitor entry [0x00007f9819560000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at java.util.Hashtable.get(Hashtable.java:356)
    - locked <0x0000000680038b68> (a java.util.Properties)
    at java.util.Properties.getProperty(Properties.java:951)
    at java.lang.System.getProperty(System.java:709)
    at com.MyClass.myMethod(MyClass.java:344)

I want to know what does the "waiting for monitor entry" state means? And also would appreciate any pointers to help me debug this issue.

Daniel Serodio
  • 4,229
  • 5
  • 37
  • 33
peakit
  • 28,597
  • 27
  • 63
  • 80
  • 4
    I would cache lookups of System properties rather than calling them repeatedly like this. You shouldn't need to call System.getProperty() more than about a dozen times over the life of the application. i.e. you should code it so its not a bottle neck. – Peter Lawrey Jul 05 '12 at 16:06

4 Answers4

6

One of your threads acquired a monitor object (an exclusive lock on a object). That means the thread is executing synchronized code and for whatever reason stuck there, possibly waiting for other threads. But the other threads cannot continue their execution because they encountered a synchronized block and asked for a lock (monitor object), however they cannot get it until it is released by other thread. So... probably deadlock.

maestr0
  • 5,318
  • 3
  • 28
  • 27
2

Please look for this string from the whole thread dump

- locked <0x00007f9819560000>

If you can find it, the thread is deadlock with thread "tid=0x00007f97f1918800"

bobon
  • 21
  • 1
  • yes bobon, I searched in the entire thread dump for this string and could not find any other reference for this id apart from the thread highlighted in the question. – peakit Jul 05 '12 at 15:33
1

Monitor = synchronized. You have lots of threads trying to get the lock on the same object.

Maybe you should switch from using a Hashtable and use a HashMap

ControlAltDel
  • 33,923
  • 10
  • 53
  • 80
  • If you see I am not using `Hashtable` directly. It's coming from my call to `System.getProperty()`. Is there an non-blocking version of `System.getProperty()`? Thanks! – peakit Jul 05 '12 at 14:54
1

This means that your thread is trying to set a lock (on the Hashtable), but some other thread is already accessing it and has set a lock. So it's waiting for the lock to release. Check what your other threads are doing. Especially thread with tid="0x00007f9819560000"

mprivat
  • 21,582
  • 4
  • 54
  • 64
  • Interestingly I do NOT see any thread with `tid=0x00007f9819560000` in the thread dump file. Any idea? – peakit Jul 05 '12 at 15:03
  • Mmmmh, probably is the VM monitor table lock then. Short of seeing the code, it's going to be tough. Essentially, the Hashtable is being competed for between two threads. One option might be to replace the Hashtable with a HashMap (because HashMap isn't thread safe). I know you're using Property but just copy into a map and use the map afterward. So you'll see it blow up on the contention (ConcurrentModificationException probably), or it starts working because the lock wasn't even needed. – mprivat Jul 05 '12 at 16:17