Why does Java throw java.lang.OutOfMemoryError: Metaspace but there is plenty of free metaspace?

Question

In my application I'm creating lots of java classes at runtime with javassist library. At some point a java.lang.OutOfMemoryError: Metaspace is thrown but java process monitoring (based on java.lang.management.MemoryPoolMXBean) reports that there is plenty of free metaspace. Why is that? And how to use 100% of metaspace memory pool?

I've created a minimal application that reproduces the problem https://github.com/vlkv/java_metaspace_oom

Donwload it, cd to the project dir and then execute 'ant run'. In this app, I've set -XX:MaxMetaspaceSize=100m, but OOM is thrown at some point around 23564Kb of metaspace used.

Call stack of the error is:

javassist.CannotCompileException: by java.lang.OutOfMemoryError: Metaspace
    at javassist.ClassPool.toClass(ClassPool.java:1170)
    at javassist.CtClass.toClass(CtClass.java:1316)
    at com.tradingview.Main.generateRandomClass(Main.java:53)
    at com.tradingview.Main.main(Main.java:24)
Caused by: java.lang.OutOfMemoryError: Metaspace
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:642)
    at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at javassist.ClassPool.toClass2(ClassPool.java:1183)
    at javassist.ClassPool.toClass(ClassPool.java:1164)
    ... 3 more

It throws this exception when the amount of memory requested *exceeds* the free space available. Not when the free space is zero. — user207421, Dec 07 '16 at 08:00
@EJP According to the OP, there's still some free space available when the exception is thrown — qxz, Dec 07 '16 at 08:03
@vitvlkv Have you debugged it to see where in the program the exception is coming from? Care to provide a [Minimal, Complete, Verifiable Example](http://stackoverflow.com/help/mcve)? — qxz, Dec 07 '16 at 08:08
@qxz Here is the "Minimal complete verifiable example" https://github.com/vlkv/java_metaspace_oom — vitvlkv, Dec 07 '16 at 08:13
@qxz "According to the OP, there's still some free space available when the exception is thrown" The ratio free/used is 3 to 1. It's not "some space" It's huge amount! — vitvlkv, Dec 07 '16 at 08:14
@EJP It seems like the program isn't requesting an amount of memory exceeding the free space available. The OP never mentions the free space "being zero" — qxz, Dec 07 '16 at 08:24
Hey guys. I've done some approximate measurements. One Class generated by generateRandomClass() method is about 1,5Kb. And OOM is thrown at the point when around 75Mb is not used. — vitvlkv, Dec 07 '16 at 08:28
@qxz There is exactly zero evidence here as to how much memory is being requested and whether that much is actually available. The OP is expressing surprise that OOM is thrown when space is available. I am explainiing how that happens. The relevance of your comment to mine continues to escape me. — user207421, Dec 07 '16 at 08:28
@EJP Except that the OP stated that "there is plenty of free metaspace." (plus their last comment) — qxz, Dec 07 '16 at 08:31
@vitvlkv have you tried to change the heap size of your JVM?? — J.Baoby, Dec 07 '16 at 08:33
@qxz 'Plenty' is meaningless. The question here is whether there is *enough*. — user207421, Dec 07 '16 at 08:33
@EJP "Plenty free" is definitely not "exceeds the free space available" — qxz, Dec 07 '16 at 08:35
@qxz 'Plenty' is 'definitely' whatever the OP meant by 'plenty'. Nothing else. He is now defining 'plenty free' as 'the point when free metaspace is almost zero'. — user207421, Dec 07 '16 at 09:28
@EJP Please point out where the OP says 'the point when free metaspace is almost zero' — qxz, Dec 07 '16 at 16:28
Same problem has been discussed [here](http://stackoverflow.com/questions/39084331/why-metaspace-size-is-twice-as-big-as-used-metaspace) — kuznet1, Dec 08 '16 at 10:17

score 0 · Accepted Answer · edited May 23 '17 at 12:01

Wow, I've just found an answer. The problem is that in my application I create a separate new ClassLoader for every generated Class. I've done a modification in my sample app that puts all the generated classes to one ClassLoader and after that, OOM is thrown at the point when free metaspace is almost zero. The fix is here https://github.com/vlkv/java_metaspace_oom/commit/d6bcade51f79758e2413d1852c771f163392c294 branch fix_of_the_problem

Does somebody knows about what are the limits for class loaders in java process? And how to increase them?

Also found this Why MetaSpace Size is twice as big as Used MetaSpace?

Please explain the contradiction between 'plenty of free metaspace' and 'free metaspace is almost zero'. — user207421, Dec 07 '16 at 09:30

Why does Java throw java.lang.OutOfMemoryError: Metaspace but there is plenty of free metaspace?

1 Answers1