0

I'm seeing strange behavior and I don't know how to gain any further insight into and am hoping someone can help.

Background: I have a query that takes a long time to return results so instead of making the user wait for the data directly upon request I execute this query via a Timer object at regular intervals and store the results in a static variable. Therefore, when the user requests the data I always just pull from the static variable, therefore making the response virtually instant. So far so good.

Issue: The behavior I'm seeing, however, is that if I make a request for the data just as the background (Timer) request has begun to query the data, my user's request waits for the data to come back before responding -- forcing the user to wait. It's as if tomcat is behaving synchronously with the threads (I know it's not -- it just looks that way).

This is in a Production environment and, for the most part, everything works great but for users there are times when the site just hangs for them and they feel it's unreliable (well, in a sense it is).

What I've done: Being that the requests for the data were in a static method I thought "A ha! The threads are syncronized which is causing the delay!" so i pulled all of my static methods out, removed the syncronization and forced each call to instantiate it's own object to retrieve the data (to keep it thread safe). There isn't any syncronization on a semaphore to the static variable either.

I've also installed javamelody to try and gain some insight into what's going on but nothing new thus far. I have noticed a lot (majority) of threads are in "WAITING" state but they also have 0ms for User and CPU time so don't think that is pointing to anything(?).

Running Tomcat 5.5 (no apache layer), struts 2, Java 1.5

If anyone has any idea why a simple request to a static variable hangs for longer background processes I would really appreciate it! Or if you know how I can gain insight that would be great too.

Thanks!

somecity
  • 1
  • 1
  • It does sound like there's a `synchronized` somewhere, or blocking call on your object somewhere that halts the request handling. (And if there isn't, it would not be thread safe) – nos Sep 01 '12 at 00:14
  • I agree, it looks like that but the two calls are totally separate and the only thing they have in common is one results updates a static variable while the other reads from it (and the order doesn't matter to me) and each method invoked are totally independent from each other. This is the essence of my confusion -- I can't see how, at all, two separate tomcat requests would become dependent upon each other when the only thing they have in common is one, non-synchronized (which is OK) static variable. – somecity Sep 01 '12 at 00:21
  • What type is that static variable, and what do you do with it ? – nos Sep 01 '12 at 00:21
  • It's a hash table that stores vectors of beans. I do a quick look-up on the desired vector of beans and return them to my jsp page. It's instant. – somecity Sep 01 '12 at 00:24
  • There is only one process (the one kicked off from the Timer) that updates the Hashtable so I can't see a thread issue there (I could be missing something obvious)... The Beans... That's interesting... They do read data from Singleton data services that return cached data as well but that's something I need make sure isn't the culprit (I don't believe it is but there is a possibility of it) -- thanks for that (now obvious) idea.... OK, just took a look at the bean and I don't see anything that would hang it -- it's all pretty straight-forward. – somecity Sep 01 '12 at 00:33

2 Answers2

0

One possible explanation is that the threads are actually blocking at the database level due to database locking (or something) caused by the long-running query.

The way to figure out what is going on is to find out exactly where the blocked threads are blocking. A thread dump can be produced by sending a SIGQUIT (or equivalent) to the JVM, and included stack traces for all Java thread stacks. Alternatively, you can get the same information (and more) by attaching a debugger, etcetera. Either way, the class name and line number of the top frame of each stack should allow you to look at the source code and figure out (at least) what kind of locking or blocking is going on.

Stephen C
  • 698,415
  • 94
  • 811
  • 1,216
  • Thanks Stephen for this... I'm in eclipse but the process is in tomcat. That is there is no breakpoint I can use while tomcat is not responding. Do you mean I should do a entire stack trace of each thread at the point of each call? Seems like my next problem would be trying to make sense out of all of the data. – somecity Sep 01 '12 at 00:45
  • The Eclipse debugger should allow you to examine the Tomcat server's stacks. You may have to suspend it from the debugger first. Alternatively, run Tomcat from the command line, and send it a SIGQUIT. – Stephen C Sep 01 '12 at 07:48
  • *"Do you mean I should do a entire stack trace of each thread at the point of each call?"*. Please reread what I wrote. *"Seems like my next problem would be trying to make sense out of all of the data."* Obviously (!!) you only have to do this for a typical stack. But, yes you DO need to understand enough of the data to be able to figure out what is going on. (If programming / debugging was easy, they could get burger flippers to do it on half your salary ...) – Stephen C Sep 01 '12 at 07:55
0

For those who would like to know I eventually found VisualVM (http://visualvm.java.net/download.html). It's perfect. I run Tomcat from eclipse like I normally do and it appears within the VisualVM client. Right-mouse click the tomcat icon, choose Thread Dump and, boom, I've got it all.

Thanks, all, for the help and pointers towards the right direction!

somecity
  • 1
  • 1