11

The story

My server is running with 24x2 processors, and the java heap is about 70 GB. At some point after installing a new version (version-B), I saw that Full GC is taking around 30sec (stopping all threads). After enabling the XX:+ParallelRefProcEnabled the weak ref processing went down to around 3-6 sec. But this is just a "Bend aid", not a cure.

2011-03-22T20:38:24.276+0000: 29540.794: [GC[YG occupancy: 5477281 K (7549760 K)]29540.794: [Rescan (parallel) , 0.4083780 secs]29541.203: [weak refs processing, 3.2855240 secs]29544.488: [class unloading, 0.0187270 secs]29544.507: [scrub symbol & string tables, 0.0095530 secs] [1 CMS-remark: 102801236K(114294784K)] 108278518K(121844544K), 3.7319690 secs] [Times: user=65.53 sys=0.14, real=3.73 secs]

Before version-B, (and without the ParallelRefProcEnabled flag) the weak ref processing used to take around 1.5 sec. (For roughly the same load)

The purpose:

What I'm trying to find out is, what was the change introduced version-B that caused the processing to jump from 1.5 sec to 30 sec. There are a few changes in version-B, and no real suspect involving weak-references.

I want to change my code so it would not relay on heavy weak ref processing.

Questions:

  • I would like to understand what exactly is happening in the "weak ref processing" stage, so I can look for a suspect, or justify a code rewrite. Any good resource for reading about what exactly happens in that stage?

  • What are the possible causes for long "weak refs processing"? (Number of weak references instances, number of objects holding a weak-reference, depth of reference-tree which is weakly referenced,...)

More info:

  • The CPU usage is not so high and doesn't seem to be an issue
  • The GC (that includes weak reference processing) occurs about every 8 minutes.
  • Running Java Sun, 1.6.0_20

I'd appreciate any response, Thanks, Erez.

Vladimir Ivanov
  • 42,730
  • 18
  • 77
  • 103
Erez Makavy
  • 111
  • 3

1 Answers1

1

AFAIK, The Reference processing time is the time it takes to determine whether a reference can be collected or not. This should be propoprtional to the number of WeakReferences you have.

It may be that a change in data structure is possible which reduces the number of weak references dramatically. e.g. say you have a cache using Map of WeakReferences. If this can be replaced with a WeakReference to a Map you may get the same outcome but with dramatically less references.

You may see a performance improvement with Java 6 update 24 which has a much newer JVM (about a year newer)

Peter Lawrey
  • 525,659
  • 79
  • 751
  • 1,130