I have a strange behaviour and maybe you can help me with it.
The environment is
- jdk_7u40 (Tried with jdk_7u51 with same behaviour)
- debian 6.0 (on windows I have never had this problem)
- jboss 7.1.1
- Geoserver 2.4.x (tried .3 and .4 with same result) which is based on spring framework
- other war modules (not spring-based, but geoserver has some dependencies on them)
The problem is that after a couple of hours that jboss is running, when I try to login to the web interface of geoserver (a POST to the j_spring_security servlet) it took A LOT (4-5 minutes) to land to the welcome page of the application.
Using jstack, I found that there is a thread that consumes 100% of a core for all the time, with and the process keep working here
at sun.security.provider.SHA2.lf_S(SHA2.java:162)
at sun.security.provider.SHA2.lf_sigma0(SHA2.java:171)
at sun.security.provider.SHA2.implCompress(SHA2.java:225)
at sun.security.provider.SHA2.implDigest(SHA2.java:118)
at sun.security.provider.DigestBase.engineDigest(DigestBase.java:186)
at sun.security.provider.DigestBase.engineDigest(DigestBase.java:165)
at java.security.MessageDigest$Delegate.engineDigest(MessageDigest.java:576)
at java.security.MessageDigest.digest(MessageDigest.java:353)
at java.security.MessageDigest.digest(MessageDigest.java:399)
at org.jasypt.digest.StandardByteDigester.digest(StandardByteDigester.java:979)
- locked <0x00000006f8c30bb0> (a java.security.MessageDigest$Delegate)
at org.jasypt.digest.StandardByteDigester.matches(StandardByteDigester.java:1099)
at org.jasypt.digest.StandardStringDigester.matches(StandardStringDigester.java:1052)
at org.jasypt.util.password.StrongPasswordEncryptor.checkPassword(StrongPasswordEncryptor.java:99)
at org.jasypt.spring.security3.PasswordEncoder.isPasswordValid(PasswordEncoder.java:204)
at org.geoserver.security.password.AbstractGeoserverPasswordEncoder.isPasswordValid(AbstractGeoserverPasswordEncoder.java:138)
at org.geoserver.security.password.GeoServerMultiplexingPasswordEncoder.isPasswordValid(GeoServerMultiplexingPasswordEncoder.java:75)
at org.springframework.security.authentication.dao.DaoAuthenticationProvider.additionalAuthenticationChecks(DaoAuthenticationProvider.java:64)
Some of you had a similar issue?
EDIT (with workaround)
I find out that the problem is related to the CMS garbage collector and to the increase of the permgen space.
Environment
The application server is JBoss 7.1.1 with 5 war deployed in it (Geoserver and others). There are shared dependencies among all the wars (with Geoserver too); Java is running with -XX:+UseParallelOldGC -XX:SoftRefLRUPolicyMSPerMB=36000
What happens
When a full gc is performed, the permgen space is increased a lot above the used. After that, the computation of methods in sun.security.provider.SHA2.* became very slow.
How did I solve
Moving to G1GC garbage collector solved the problem for me (currently I'm using the following options -XX:+UseG1GC -XX:-UseAdaptiveSizePolicy -XX:SurvivorRatio=1 -XX:NewRatio=1 -XX:MaxTenuringThreshold=15 -XX:G1HeapRegionSize=32m )
