9

It is Hallowe'en after all.

Here's the problem: I'm maintaining some old-ish J2EE code, using Quartz, in which I'm running out of threads. jconsole tells me that there are just short of 60K threads when it goes pear-shaped, of which about 100 (!!) are actually running. Intuition and some googling (see also here) suggest that what's happening is something (I'm betting Quartz) is creating unmanaged threads that never get cleaned up.

Several subquestions:

  1. It there a tool that I can use easily to trace thread creation, so I can be certain the issue is really Quartz?

  2. Most everything I've found about similar problems references Weblogic; is this a false lead for Tomcat?

  3. Does anyone have a known solution?

It's been years since I did J2EE, so I wouldn't be too surprised if this is something that can be solved simply.

Update: It's clearly increasing threads without bound, see this plot from jconsole.

They're dead, Jim

Glorfindel
  • 21,988
  • 13
  • 81
  • 109
Charlie Martin
  • 110,348
  • 25
  • 193
  • 263
  • Do you notice any pattern in the names of the threads? I would suspet the problem is the code that uses Quartz, not Quartz itself – matt b Oct 30 '09 at 16:25
  • 8
    +1 for the title, tells me everything I needed to know. Did you try the shotgun? – Anthony Oct 30 '09 at 16:28
  • Matt, I'm really out of date with JEE debugging tools (just asked a separate question for recommendations: http://stackoverflow.com/questions/1650881/state-of-the-art-in-j2ee-debugging-and-monitoring-tools/1650922#1650922) so I haven't yet. But there is a known Quartz bug causing it to spawn unmanaged threads. – Charlie Martin Oct 30 '09 at 17:45
  • Anthony, I'd like to try the shotgun on both application and client. Upon legal advice, I refrained. – Charlie Martin Oct 30 '09 at 17:46
  • Oh man, I know some women who'd kill for thread counts that high :P – Alex Marshall Aug 05 '10 at 23:51

2 Answers2

4

  • Try to increase the logging level of org.quartz.simpl.SimpleThreadPool to debug to get more information.

  • If that does not work, try a logging listener. Quartz has a JobListener interface, which is specified in its tutorial. A listener can help you trace job execution. Maybe jobs just don't finish and get deadlocked.

  • Configure org.quartz.threadPool.threadCount to stop running out of threads.

update:

  • Also, you might want to take a thread dump and see the thread stats. visual vm has a plugin called TDA, or you can use Thread dump analyzer directly.

  • Just in case, check the quartz version to see if there is no known bug.

HMM
  • 2,987
  • 1
  • 20
  • 30
0

Have you had a look with jvisualvm - it gives some more information.

Also, get stack traces to see what the threads are actually waiting on. You might have an aha-feeling right there.

Thorbjørn Ravn Andersen
  • 73,784
  • 33
  • 194
  • 347