2

We run a cluster of ColdFusion 8 servers on top of the Sun JVM, version 1.6.0_11. Occasionally a server will simply stop responding to requests. We've determined so far that the JVM is locking up and that neither ColdFusion nor IIS are the root of the problem. I've included our JVM arguments below. Any additional JVM settings we should be looking at? Any tools to monitor, trace, or get metrics from inside the JVM so we can see what it's doing when the crashes occur? Any other JVM troubleshooting tips?

-server
-Xms1024m
-Xmx1024m
-Dsun.io.useCanonCaches=false
-XX:MaxPermSize=256m
-XX:PermSize=256m
-XX:+UseParallelGC
-Dsun.rmi.dgc.client.gcInterval=300000
-Dsun.rmi.dgc.server.gcInterval=300000
-Djmx.invoke.getters=true
Justin Scott
  • 8,798
  • 1
  • 28
  • 39
  • What does the thread dump say. Without knowing your application I can suggest that one of the most likely causes of apparent lockup will be exhaustion of the database connection pool assigned to the application. This is generally pretty easy to detect from the thread dump – Dave Cheney May 26 '09 at 15:49

5 Answers5

2

This is a standard option for my config:

-XX:+HeapDumpOnOutOfMemoryError

I've also have SNMP/RRD monitoring of basic JVM health characteristics (like heap size, thread count)...there are more.

Then, there is the entire world of tools like jconsole...

Stu Thompson
  • 3,349
  • 7
  • 31
  • 47
  • Problem is that this will sometimes consume minutes of time while the server is offline. If you monitor the server and restart it if it stops responding then you might kill the JVM dump before its finished dumping. – djangofan Jun 09 '11 at 17:25
1

If you are on Linux you could use kill -3 to get a thread dump and look for deadlocks using using Thread Dump Analyzer.

VisualVM now ships as part of the JDK and can be used monitor memory usage, threads and also includes a profiler.

jmap can be used to obtain heap histograms and heap dumps from Java processes. You could then use a tool like Eclipse Memory Analyzer to investigate it.

As another answer mentioned you could look into JMX and JConsole. For each service in our application we have a MBean so we can view keys stats such as queue sizes to check if anything seems out of the ordinary.

Mark
  • 1,331
  • 1
  • 11
  • 16
  • The very recent JDK versions have a better edition of the VisualVM than the older ones. Take care that you have a recent JDK. – djangofan Jun 09 '11 at 17:24
0

you can use jstat to see the heap usage.

jstat -gc -h10 -t 1000

0

You need to look into JMX monitoring - it can give you good insight as to what's going on inside the JVM.

To start, add -Dcom.sun.management.jmxremote.port=portNun to the list above. Then, on another machine, run jconsole and tell it to connect to the machine and port you specified above. Poke around at will and see what threads are running and what's going on after it's stopped responding to requests.

pjz
  • 10,595
  • 1
  • 32
  • 40
0

jvmstat has been helpful to me in the past, though I don't think I used it with CF 8/JRE 1.6. It's got nice displays of what's in each generation so you can see if some of your memory variables are causing the problem.

Milner
  • 935
  • 7
  • 17