In a nutshell: my JBoss instance is running ok, but after some days it's performance is slowly degrading.
Detailed: I've got a setup with JBoss 5.1.0-GA and Java 1.6.0_18-b07 (x64) running on a 64 bits RHEL 4 box. The hardware is a virtual machine with 8 core Xeon X5550 / 20G ram.
The product deployed in JBoss contains a webservice on which a endurance test is performed. No database is involved in the process.
The tests are performed using soapui with 4 threads and the tests are configured to create 20% cpu usage.
Let say, at first the average response times are 300ms. After 2 days, the response times are now 600ms, which I don't understand. Of course I did some checks:
- There are no memory leaks (confirmed with jprofiler)
- Heap mem is always around 25-50%, perm space usage is 50%
- GC is almost never busy
- All threads are idle after inspecting a thread dump
While do some further investigations, I did a cpu profile with JProfiler at the beginning (when it's still fast), and on on the (slow) end. What I see then, is that every single call just is 100% slower! Even call's to a simple Map#put(). (the # of invocations and the content of these maps are the same). When running a profiler, there are no signs of blocked threads, just running threads.
Does anyone has a clue what's causing the performance degradation? Thanks!
Update: solved the performance degradation by upgrading the Java version to 1.6.0_24 !
While out of options, I scanned through all the release notes of the java vm, and discovered a performance and reliability fix in 1.6.0_23. See also the 1.6.0_23 release notes
After the jvm upgrade, the performance stays the same and does not degrade over days.
