I'm diagnosing a customer issue that crops up very occasionally: ClassNotFoundExceptions for classes compiled from JSPs. Most of the time everything works fine, but every so often (maybe once every couple of weeks across a dozen-ish nodes) Tomcat can't load the class file for a particular JSP (generally just one page, seemingly at random). The class file exists in the 'work' directory. It doesn't appear to be corrupt. I can remove it and the corresponding .java file, and it will be recreated (identically) on the next request - but Tomcat continues to report the ClassNotFoundException. Restarting Tomcat fixes the problem.
Platform details:
- RedHat Enterprise Linux (RHEL) 6
- Tomcat 6.0.24-72 (official 'tomcat6' package installed with yum)
- OpenJDK Java 1.6.0_32
I have been digging through source code for the Jasper JSP compiler and the JVM classloader code that it uses to load these classes (org.apache.jasper.servlet.JasperLoader based on a java.net.URLClassLoader). If I'm reading the code correctly, each JSP page gets a corresponding 'context' object (JspCompilationContext/JspServletWrapper) created when a particular JSP is first accessed. The JspCompilationContext contains the JasperLoader associated with the page.
My best theory is that something within the classloader is caching a reference to, e.g., a filesystem inode that becomes invalid (e.g., due to a filesystem maintenance activity). However, I can't see anything in the classloader code that would be doing that, and it looks like a new classloader should be created if the JSP is recompiled, and such recompilation doesn't fix the issue. So, my best theory is not good.
Unfortunately this has only been seen at a customer site, and is not reproducible at will, so my ability to debug directly is limited. As such, I'm looking for any good possibilities to investigate.
