2

we have a situation wherein the metaspace of a springboot microservice keeps growing but the heap is behaving well.

jmap -clstats shows there are thowsands of dead classloaders of the following kind.

0x00000000e3c8e3c8 1 4087 0x00000000e0004d18 dead com/sun/org/apache/xalan/internal/xsltc/trax/TemplatesImpl$TransletClassLoader@0x000000010087c7e8

At the initial high watermark GC is being triggered and I see a drop in metaspace. After this forced GC that is triggered due to the defined metaspacesize, I see metaspace is continuously growing and I see more dead classloaders of the same kind are being kept in metaspace. I do see a bit of GC activity but no drop in Metaspace consumption. However, if I force GC collection through visualvm, ton of classes are being unloaded and metaspace consumption is going back to the state when the service was started.

Why would JVM managed GC not unload these dead classloaders but a forced GC does? If weak/soft/phantom reference is the reason then shouldn't that apply to forced GC as well?

This is on Java8. Can anyone give some pointers as to where I should look next? Obviously there is a leak so is there a way to know the parent classloader of the TemplatesImpl$TransletClassLoader?

Appreciate any help.

Vijay Chavva
  • 21
  • 2
  • I encounter the same problem but I am using jdk7. https://stackoverflow.com/questions/67133935/jvm-full-gc-cant-unload-classes-even-permgen-is-full – Xinli Niu Apr 18 '21 at 16:01

1 Answers1

0

First thing is that, JVM will clear Metaspace ONLY during Full GC and NOT during Young GC. So the behavior you are seeing is expected.

At the initial high watermark GC is being triggered and I see a drop in metaspace.

If you check GC trace, you will see a System.GC() call. That's Full GC.

However, if I force GC collection through visualvm, ton of classes are being unloaded and metaspace consumption is going back to the state when the service was started.

Again, that will be a Full GC triggered by visualvm and can be seen in GC trace. That is why you are seeing the utilization drop.

From the description you provided, I don't think you have a classloader leak because everything is getting cleaned up during Full GC. From my experience, I would only consider something as a "memory leak" if app is unnecessarily keeping some objects alive which therefore is not getting garbage collected - even during Full GC. In your case I will suggest to limit Metaspace size by using -XX:MaxMetaspaceSize flag. When occupancy will reach that threshold, JVM will automatically trigger a Full GC and as you noticed, Metaspace usage will drop. Set that limit judiciously because too low value will cause java.lang.OutOfMemoryError: Metaspace issue.

Some more details on Metaspace can be found here.

suv3ndu
  • 221
  • 5
  • 12
  • Could you please help to check my problem? The classloaders are dead and can be recalimed by visualvm click. But it went oom if I dont trigger the gc through visualvm. https://stackoverflow.com/questions/67133935/jvm-full-gc-cant-unload-classes-even-permgen-is-full – Xinli Niu Apr 18 '21 at 16:03