0

I have a Java VM that is hanging "randomly". I quote the randomly bit, because there is obviously a reason that the VM is hanging, but the hang does not occur periodically. We have the same software running in different customer environments and in those environments the JVM is not hanging.

In the process of attempting to troubleshoot the hang the process exists with zero CPU utilization. I then attempt to execute kill -3 and the kill command hangs. No JVM Thread Dump is produced.

I have spent time instrumenting the code to periodically log the thread stack traces hoping to catch the JVM in a state that would indicate where the issue lies, but so far this attempt has not born much fruit.

Unfortunately I have not been able to reproduce this issue in my lab environment so I am limited by what can be done at the Customer site.

The OS's in question are Red Hat Enterprise 5.4 and SUSE 10 running java version 1.6.0_05-b13

Has anyone had this problem? Any ideas on why kill -3 is failing to produce a Java Thread Dump?

Thanks!

Hans
  • 1
  • 1
  • 1

4 Answers4

1

Sounds like you might be running out of heap space.

Even if your JVM does not produce thread dumps, you should be able to generate a heap dump. This, with the GC stats, will also give you insight into what is going. Consult your JVM's documentation for how to generate heap dump in binary format. Then download Memory Analyzer and open the heap dump file to see what is taking up memory. Typically, one would use the jmap command with the proper options. Generating the heap dump will cause your JVM to slow down quite a bit; I've seen it take 30 mins to generate a heap dump 1.5GB large, YMMV.

Also, post your java opts, both from the servers where it is hanging and working fine.

KM.
  • 1,786
  • 2
  • 18
  • 31
0

Have you tried jstack or jvisualvm?

Paul Tomblin
  • 5,225
  • 1
  • 28
  • 39
  • Yes I have tried jstack, but not jvisualvm. Basically anything that attaches to the JVM will hang when the JVM gets in the hung state. – Hans Apr 08 '11 at 23:45
0

Are any core files created when the JVM crashes?

You can also write GC statistics to a log file. This may provide some more information as to why the JVM is hanging.

Are there any other JVM's using that version of Java running on that machine? My first assumption is that there is something wrong with the JRE, and it may need to be re-installed.

Hope that helps!

Tom Purl

Tom Purl
  • 549
  • 1
  • 3
  • 13
  • The JVM does not crash, kill -9 is not causing a core dump. I have not considered turning on garbage collection statistics, I will have to see how large this log file can grow. The JVM doesn't hang everyday and can go a couple of weeks before hanging. I am working with a single system, but many other systems are hanging as well in the same customer environment. The same systems in other customer environments are not hanging. I don't think it is a JRE issue. Thanks for the GC suggestion though. – Hans Apr 08 '11 at 23:41
0

Are you starting your jvm (or container with java options) with the -Xrs switch? I thought that this command line option would prevent a kill -3 from working. If so, you can get a thread dump using jstack if you are using a 1.5 or higher jvm.

mahnsc
  • 1,796
  • 13
  • 11
  • We are not using the -Xrs switch. I can cause a thread dump from the jvm when the system is running normally. I did try jstack and I get the same result. Jstack hangs as well. Thanks for the suggestions – Hans Apr 08 '11 at 23:39
  • I suppose another possibility is that the JVM is paused at this time due to Full GCs and Tom Purl's suggestion above about writing garbage collection logs would let us know for sure. – mahnsc Apr 10 '11 at 01:31