I am using VisualVM to monitor my JBoss instance. I have attached a screenshot of it aswell.
The problem is after I restart the JBoss instance, the CPU on the OS starts to go high. Load can go as high as 40 and JAVA process in top command shows upto 300% usage.
This then goes on to slow down the application at the front end.
VisualVM shows that CPU is high and that thread count is increasing also.
How can I further go to the root cause of this ?
A good start point would be to generate a Heap Dump, and use a Heap Dump analyzer tool to see what is happening under the hood, and browse it with VisualVM
Some free alternatives for you are Eclipse MAT or IBM HeapAnalyzer
When it comes to high CPU usage, I generally look for two things :
To diagnose #1 further, enabling GC Logging may be the best way. However, you can use jstat as well:
jstat -gc PID 60s
The above command will collect the memory usage and garbage collection details from the JVM every 60s. In a debugging environment, 60s should be OK, but in a production environment 3600s should be sufficient. jstat being very light weight, doesn't create any adverse impact. The output will help you to understand if there are frequent GCs (major/minor). Frequent major collection is a problem for sure (it pauses the application), however, very frequent minor collection could also cause high CPU (application is creating too much of garbage too frequently). If this is the case, probably you need a head dump and you need to understand application's memory usage details, but not before that. Please remember, capturing heap dump may "hang" your application (I don't suggest it in production unless you can restart the application immediately after capturing the data).
To diagnose #2, "top" provides an option ("H") to check the CPU consumed by individual threads. That will point (in real time) the application threads which are consuming CPU. Also (as other's have suggested), capture 5/6 thread dumps in an interval of 10 seconds each. Looks for threads in RUNNABLE state. These are the threads which are doing work and hence occupying CPU. Is the same (set of) thread stuck in the Runnable state over multiple dumps? Probably you have got a problem.
Hope this helps. Best of luck!
Understanding increasing thread count is fairly easy. Capture several theaddumps and compare what new threads are created.
For high CPU utilization you have to detect what threads are causing this, either using Visual VM (if this information is available) or native system monitoring tools. You should be able to map the thread ID to the stacktrace in threaddump to see the real cause.
Yes but it is extremely difficult for us to find from the thread dump because of 2 issues. 1. The CPU utilization is increases rapidly from 20% to 90% in 15-20 mins. We are not able to do so in other environments. 2. By the time we take the thread dump the actual thread which might be causing issue get completed and we find other threads which are waiting for resources.
Is there any other way which can be configured or checked on server side ?
