100 threads TIMED_WAITING in tomcat, causing it to stall as total number of threads crosses 200

Question

Recently one of our production tomcat server became unresponsive because tomcat's busy threads shot upto 200. When we took thread dump before restarting we got 100 threads in TIMED_WAITING state like these 3 threads:

""http-bio-7007"-exec-241" daemon prio=10 tid=0x00002aaab107b000 nid=0x59df waiting on condition [0x0000000051239000]
java.lang.Thread.State: TIMED_WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x0000000580d877d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
   at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424)
   at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)
   at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:662)

""http-bio-7007"-exec-237" daemon prio=10 tid=0x00002aaab186e000 nid=0x596d waiting on condition [0x000000004d1f9000]
java.lang.Thread.State: TIMED_WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x0000000580d877d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
   at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424)
   at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)
   at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:662)

""http-bio-7007"-exec-236" daemon prio=10 tid=0x00002aaab1118000 nid=0x596c waiting on condition [0x000000004e50c000]
java.lang.Thread.State: TIMED_WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x0000000580d877d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
   at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424)
   at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)
   at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:662)

We have 4 application's thread pools(e.g. pool-4-thread-20 etc) also which are having 20 threads each so I am not sure on which blocking queue these 100 threads waiting? We are using c3P0 connection pool with hibernate which doesn't seems to be cause of this.

Any idea what java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject is?

Did you try taking a Heap Dump and running it through the [MAT](http://www.eclipse.org/mat/) to see what Objects are accumulating ? — Bruno Grieder, Jun 04 '12 at 15:04
At the moment we have the same problem. Even restarting tomcat didn't help. After rebooting evrything worked again. STRANGE! ill investigate further and will report if I find something interesting. — Janning Vygen, Jul 01 '12 at 09:06
It has nothing to do with hibernate as we encountered this problem tonight across our complete server farm and some of them are just image servers without any hibernate or database stack. — Janning Vygen, Jul 01 '12 at 09:07
Our problem was due to leap second bug. rebooting the server fixes it. some report restarting ntp would help too. see here http://serverfault.com/questions/403732/anyone-else-experiencing-high-rates-of-linux-server-crashes-during-a-leap-second — Janning Vygen, Jul 01 '12 at 11:41
This got fixed when we fixed our code which was leaking DB connections managed by c3p0. There were few flows in our code where we were not calling rollback() specifically in catch block before closing entity Manager in finally block, so in case of exceptions connection was not coming back to pool and if frequency of exception is high(more than size of pool within timeout interval) then all other process threads would pile up to get connection. — ThinkFloyd, Jul 04 '12 at 17:56
This is such a badly written question and will surely confuse many people reading it and the answers associated with it. TIMED_WAITING (parking) is exactly what you would see on threads that are waiting for connections. I wish there was a way this question could be deleted to save considerable wasted time for however many people end up reading it. — stepanian, Aug 10 '14 at 02:53

score 11 · Accepted Answer · answered Nov 30 '12 at 05:53

11

This got fixed when we fixed our code which was leaking DB connections managed by c3p0. There were few flows in our code where we were not calling rollback() specifically in catch block before closing entity Manager in finally block, so in case of exceptions connection was not coming back to pool and if frequency of exception is high(more than size of pool within timeout interval) then all other process threads would pile up to get connection.

answered Nov 30 '12 at 05:53

ThinkFloyd

4,981
6
36
56

I am getting the same issue, can you please assist me on how to make use of connection ? So that it can't get leak. – Mohanraj Apr 21 '16 at 07:11

score 5 · Answer 2 · edited Aug 17 '12 at 22:02

5

Any idea what java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject is?

The ConditionObject is used inside the queue to synchronize access to the queue by different threads.

Its the standard stacktrace, when a thread of your executerpool is idle and waiting for new tasks.

edited Aug 17 '12 at 22:02

nhahtdh

55,989
15
126
162

answered Jul 30 '12 at 06:54

Mirko

1,512
1
12
19

100 threads TIMED_WAITING in tomcat, causing it to stall as total number of threads crosses 200

2 Answers2

Linked