Coldfusion Metaspace null

Question

Working with CF2016 I have been seeing this error in a developer environment and also in production enviroments but less frecuently.

Metaspace null. The error ocurred on line -1.

Reading some forums get the idea of the garbage collector and managed to implement this code to force the collector:

<cfset runtime = CreateObject("java","java.lang.Runtime").getRuntime()>
<cfset freeMemory = runtime.freeMemory() / 1024 / 1024>
<cfset totalMemory = runtime.totalMemory() / 1024 / 1024>
<cfset maxMemory = runtime.maxMemory() / 1024 / 1024>
<cfset usedMemory =  (runtime.totalMemory() - runtime.freeMemory()) / 1024 / 1024>

<cfoutput>
    Used Memory: #Round(usedMemory)#mb<br>
    Free Allocated Memory: #Round(freeMemory)#mb<br>
    Total Memory Allocated: #Round(totalMemory)#mb<br>
    Max Memory Available to JVM: #Round(maxMemory)#mb<br>
</cfoutput>

<cfset clear = runtime.gc()>

Curiously, the runtime memory has a lot of space (normaly half of space is free) But Metaspace Null still appearing.

Another aproache could be the loaded classes, maybe they are not being destroyed automatically. But I can't find a way to see the loaded ones.

So far the "solution" is to restart the server so I can operate normally. I was able to reduce the frecuency of the error increasing the Maximum JVM Heap Size to 1024MB but still ocurring

And the stacktrace... well is hard to get to the line -1 in a ghost file! xD

Java Version: 1.8.0_112

Garbage Collector: -XX:+UseParallelGC (default)

Arguments to VM (line jumps for easy reading)

java.args=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5006 -server 
-Xms256m 
-Xmx1024m 
-XX:MaxMetaspaceSize=192m 
-XX:+UseParallelGC 
-Xbatch 
-Dcoldfusion.home={application.home} 
-Djava.awt.headless=true 
-Duser.language=en 
-Dcoldfusion.rootDir={application.home} 
-Djava.security.policy={application.home}/lib/coldfusion.policy 
-Djava.security.auth.policy={application.home}/lib/neo_jaas.policy 
-Dcoldfusion.classPath={application.home}/lib/updates,{application.home}/lib,{application.home}/lib/axis2,{application.home}/gateway/lib/,{application.home}/wwwroot/WEB-INF/cfform/jars,{application.home}/wwwroot/WEB-INF/flex/jars,{application.home}/lib/oosdk/lib,{application.home}/lib/oosdk/classes 
-Dcoldfusion.libPath={application.home}/lib 
-Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true 
-Dcoldfusion.jsafe.defaultalgo=FIPS186Random 
-Dorg.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.JavaUtilLog 
-Djava.util.logging.config.file={application.home}/lib/logging.properties

1. What is the version of Java that you are running? 2. And what GC are you using? 3. I have been down this road that you are going. It is possible for ColdFusion to be able to create and destroy variables faster than the JVM can keep up. In my particular case, I was using Query of Queries inside of loops. In my opinion, you should find the code that is causing the problem, rather than forcing a gc. — James A Mohler, Apr 17 '18 at 17:05
@JamesAMohler Check the update please. About the code, yes, I use QoQ in some parts but this error is thrown at any moment or any process, not in a particular one. After restart, the process that cause the error works properly, a few hours or days later fails again in any other process. Thats why I haven't being able to identify code. — Giancarlo Benítez, Apr 17 '18 at 17:45
Sort answer, consider using G1GC. Long answer can be seen here: https://stackoverflow.com/questions/20253684/coldfusion-garbage-collection — James A Mohler, Apr 17 '18 at 22:59
@GiancarloBenítez - Did you check the various log files and do they contain the error? — SOS, Apr 17 '18 at 23:57
@Ageax I have read the log files (all of them) but no luck finding the error. Maybe I'm not reading the correct one or missed something, any suggestion? — Giancarlo Benítez, Apr 18 '18 at 14:18
(Edit) @GiancarloBenítez - I guess it's possible it wasn't logged, but what did you search for in the *.log files - the word "Metaspace" only? I know they replaced PermGen with Metaspace in jvm 1.8, so I was curious if there was any sort of OOM error in the logs anywhere. Finding it in the logs isn't going to solve the problem, but would confirm it's a memory issue and maybe provide more details. BTW how much memory is available? -Xms256m -Xmx1024m isn't a huge amount — SOS, Apr 18 '18 at 14:39
@Ageax With "metaspace" was the only word I got some results from de "application.log" but just show the route of the file that failed, nothing more. "OOM" or "PermGen" gave nothing. About memory, this is a laptop local instance for development. The production serv has about 6GB and still having errors. — Giancarlo Benítez, Apr 18 '18 at 22:40

score 3 · Answer 1 · answered May 08 '18 at 20:53

Your maximum metaspace size is set to 192MB - I'd up this to at least 512MB by updating the following JVM flag:

-XX:MaxMetaspaceSize=512m

To keep your 192MB initially and try to save (meta)space, also add:

-XX:MetaspaceSize=192m

This fixed our issue when we were experiencing the same as you. Additionally you can run your garbage collection inside a scheduled task which can run once daily if you want to make sure it's happening regularly.

charlie arehart · Answer 2 · 2020-02-27T20:42:12.330

I'll offer a different perspective: for most CF folks who hit such a metaspace error as above (or "outofmemoryerror: metaspace"), it's nearly always because Adobe sets that relatively low default 192mb value shown, by default.

And while raising the value is indeed one solution, I argue that the best solution for most folks is simply to remove the maxmetaspacesize argument. I explain more, including why Adobe sets it (a remnant of an old situation related to maxpermsize from Java 7 and earlier) here:

https://www.carehart.org/blog/client/index.cfm/2020/2/24/solving_metaspace_errors/

Coldfusion Metaspace null

2 Answers2