0

I have a J2EE java application which processes SOAP requests. In our production environment (HPUX,OC4J,Java 5) we have about 20 threads running for this process, and we sometimes see 1 thread pausing for ~15 seconds. Until now, I haven't succeeded replicating the problem in our preproduction environment, and I'm scared of breaking stuff and violating SLA's if I use jconsole and associated tools on our production server.

Who has any inspiration? I know about http://java.sun.com/j2se/1.5/pdf/jdk50_ts_guide.pdf but I miss the experience to dare using it straight in production (plus, the HPUX guys threw some of these tools out of the toolbox, replacing them with HPJMeter)

Also, although this suggests a GC problem to me, I don't yet know enough to prove or disprove this theory and I am open to other suggestions.

j0k
  • 22,600
  • 28
  • 79
  • 90
hyperman
  • 1,324
  • 1
  • 11
  • 21
  • 1
    GC shouldn't be a problem here, since if it would suspend a thread (a stop-the-world gc) all other threads should be affected as well. – Thomas Sep 07 '11 at 08:27
  • Apart from the fact that I don't necessarily see a problem connecting with jconsole, have you analysed thread dumps? What was the thread doing when he paused? On unix machines, thread dumps can be obtained when running `kill -3` on the JVM process – Lukas Eder Sep 07 '11 at 12:31

2 Answers2

1

We connect jconsole (and other tools) straight to production regularly. There is no significant overhead for us, the instrumentation is already going on within the JVM so you'd just be connecting a remote process to read published values. I say go for it!

Either way, you really need to see what's going on on the box. Thread dumps might or do some internal instrumentation. By internal instrumentation, I mean recording key measures within the code and exposing those somehow. It's essentially what the JVM does (exposing them via JMX) but rolling your own gives you more specificity. For example, I'm frequently recording request/response or other critical path performance timings internally.

oh, and one more thing. You can setup your app to using an agent to provide even more information. Typically this would be to plug a profiler in (like jprofiler or yourkit) but this does usually add more overhead and isn't recommended for production.

It's also worth thinking about the cost of not getting the information you need out of the VM. For example, is the cost of not fixing the issue more or less than the cost of a small % drop of performance when monitoring?

More scientifically, this article has some comments. It's suggesting up to 7% overhead (contradicting my previous point), a previous article from 2006 suggests 3-4% but both are highly contextual results. For example, CPU intensive applications may or may not be affected more than IO bound ones.

So a more appropriate answer from me (rather than just "go for it") would be to understand the impact it would have for your application in your environment through measurement. Run representative tests on a similar environment to production with jconsole connected and disconnected and see for your self.

Also see this stackoverflow question.

Community
  • 1
  • 1
Toby
  • 9,523
  • 8
  • 36
  • 59
  • If some of you guys could second the 'go for it, there's no real risk', i'll try. *We've made some thread dumps, but they don't really show anything unexpected. Also, the problem is intermittent and we have no thread dump when the problem actually occurs. *What do you mean by 'internal instrumentation'? I can't change the production code for now if that's what youy mean. If some of you – hyperman Sep 07 '11 at 11:26
  • dynaTrace is another really good product. It has some overhead, but helps avoid it by working directly with the bytecode. – Chris Aldrich Sep 07 '11 at 12:53
1

There are a few things that you can do on HP-UX to get additional information from a running Java process. If you send the PROF signal to the JVM, it will toggle the generation of a GC log (as if you had used the -Xverbosegc command line option). Generating the GC log is very inexpensive, so you should be able to turn this on in production without affecting the performance.

If you send the USR2 signal to the JVM, it starts profiling (same as -Xeprof). If you send the signal a second time, it turns off the profiling. This will have a noticeable performance impact, though it is smaller that what you would see from an external, third party profiler.

You can analyze the resulting data files using HPjmeter. HPjmeter can also connect to a running JVM for real-time monitoring. With Java 5, you need to start the JVM with the -agentlib option. If you were using Java 6, you could attach to the running JVM without needing any extra command line options.

JZeeb
  • 321
  • 1
  • 3