What do the "compacting perm gen" values represent?

Question

I'm investigating a JVM crash on one of our production systems, what do the following memory values represent in the hs_err_pid log file snippet below?

Heap
 par new generation   total 1258624K, used 955445K [0x00000005c0000000, 0x00000006155b0000, 0x000000066aaa0000)
  eden space 1118784K,  73% used [0x00000005c0000000, 0x00000005f1e52598, 0x0000000604490000)
  from space 139840K,  98% used [0x000000060cd20000, 0x00000006153db100, 0x00000006155b0000)
  to   space 139840K,   0% used [0x0000000604490000, 0x0000000604490000, 0x000000060cd20000)
 tenured generation   total 2796224K, used 1745107K [0x000000066aaa0000, 0x0000000715550000, 0x00000007c0000000)
   the space 2796224K,  62% used [0x000000066aaa0000, 0x00000006d52d4d90, 0x00000006c2e0c400, 0x0000000715550000)
 compacting perm gen  total 482944K, used 482943K [0x00000007c0000000, 0x00000007dd7a0000, 0x0000000800000000)
   the space 482944K,  99% used [0x00000007c0000000, 0x00000007dd79fff0, 0x00000007dd7a0000, 0x00000007dd7a0000)
No shared spaces configured.

My concern is with the "compacted perm gen" usage: does it mean percentage used of maximum allocated perm gen heap, or percentage used of maximum heap, or something else? The percentage provided appears to be a division of the used/total, is this the total allocated perm gen? Since our -XX:MaxPermSize is set to 1GB...

Are there any useful resources (other than the Oracle whitepaper, which does not mention hs_err files) to interpret the data dumped on a JVM crash?

If you make the division, you'll get it. The percentage of the permgen used. (11253K / 16384K) — zeller, Mar 25 '13 at 09:28
Sorry, my example was not a good representation of the error we're actually seeing. I've updated my question. — seanhodges, Mar 25 '13 at 10:01

score 5 · Accepted Answer · answered Jun 03 '13 at 09:41

I never found a reference that accurately described the "compacting perm gen" values, but our own investigation proved that the reported values were:

currently used PermGen / currently allocated PermGen

In the example in my question, this meant that 482944K of memory had already been allocated for PermGen and 482943K of it had been used at the point of GC (99%). Our max PermGen size was set to 1048576K (1GB), so the collection process had plenty of reserved resources to re-allocate with.

For those encountering similar issues - we solved our problem in the end. In our case it turned out to be a third-party library that made use of the sun.misc.Unsafe class, which is notoriously "unsafe" when used incorrectly.

In this case, a piece of logic for cloning objects passed a specific ClassLoader to some sun.misc.Unsafe operations to copy objects. On certain machines, the copied objects were regularly being created in a corrupted state. When the JVM attempted to garbage collect, it would eventually harvest one of these bad objects and crash. This always caused the error described in my question.

I guess in the end it had nothing to do with compacting PermGen. I assume by "compacting" it really means garbage collection / unload classes in PermGen. I did a quick test and it's indeed the currently allocated PermGen and not the max perm space. CGLib has also been reported by others to stuff PermGen in the past, but since Hibernate 3.3, it's switched to Javassist which seems to have alleviated this exact issue. — Christopher Yang, Jun 03 '13 at 11:19
Thanks for confirming this, as a result I've marked my question as solved. — seanhodges, Jun 03 '13 at 11:28

Christopher Yang · Answer 2 · 2013-06-03T06:53:40.120

0

But just because you set MaxPermSize to 1GB doesn't mean HotSpot will honor it. I don't think I've ever gotten more than 512MB. Your JVM's 482MB is pretty close to this figure.

In any case, 512MB is plenty for PermGen since it's only used to hold classes' metadata, it won't grow in size unless more classes are loaded through ClassLoader. So the question is: do you really need more than 512MB to hold roughly the size of the bytecodes of all the necessary classes in memory?

It's very possible that your system actually has memory leaks in PermGen.

edited Jun 03 '13 at 06:53

answered Jun 03 '13 at 06:45

Christopher Yang

3,769
4
30
27

Unfortunately not. This is a high traffic site using Hibernate - so lots of CGLIB class proxies, hence the large PermGen values. Several days of uptime did not indicate a constant growth of any heap usage. We did solve the problem recently, I'll post my answer shortly. – seanhodges Jun 03 '13 at 08:39

What do the "compacting perm gen" values represent?

2 Answers2

Linked