JVM hangs because of infinite GC

Question

I have a huge application running in glassfish server which creates lot of short-living objects and i have the following GC configuration in JVM.

-XX:+DisableExplicitGC
-XX:+UseParallelGC
-XX:+UseParallelOldGC
-XX:-UseAdaptiveSizePolicy
-XX:PermSize=256m
-XX:MaxPermSize=1024m
-Xms7g
-Xmx7g
-XX:NewRatio=2

But the JVM is hanging with Infinite GC. I have to restart the JVM. I am getting the following info from GC Log.

2.855: [GC 734029K->9736K(7034240K), 0.0133500 secs]
2.869: [Full GC 9736K->9501K(7034240K), 0.1043570 secs]
13.254: [GC 681231K->26506K(7034240K), 0.0251050 secs]
13.280: [Full GC 26506K->26082K(7034240K), 0.2904930 secs]
13.589: [GC 103156K->26224K(7034240K), 0.0015940 secs]
13.590: [Full GC 26224K->24440K(7034240K), 0.2254710 secs]
35.478: [GC 1859512K->131673K(7034240K), 0.0781300 secs]
41.603: [GC 1966745K->351954K(7034240K), 0.1858590 secs]
46.012: [GC 2187026K->502362K(7034240K), 0.2329020 secs]
51.850: [GC 2337434K->608654K(7034240K), 0.2012410 secs]
72.584: [GC 2443726K->727923K(7034240K), 0.2203390 secs]
80.239: [GC 2562995K->894770K(7034240K), 0.2323490 secs]
106.221: [GC 2729842K->1265916K(7034240K), 0.2800630 secs]

Please let me know whether the jvm GC settings are right for this use case. Or Any help to resolve this issues is much appreciated.

Update I also got jmap heap dump info. PS Old generation seems to hold most of the memory even when nobody is using it. It is not increasing(which will in case of memory leak).

using thread-local object allocation.
Parallel GC with 8 thread(s)

Heap Configuration:
   MinHeapFreeRatio = 40
   MaxHeapFreeRatio = 70
   MaxHeapSize      = 7516192768 (7168.0MB)
   NewSize          = 5439488 (5.1875MB)
   MaxNewSize       = 17592186044415 MB
   OldSize          = 5439488 (5.1875MB)
   NewRatio         = 2
   SurvivorRatio    = 8
   PermSize         = 268435456 (256.0MB)
   MaxPermSize      = 1073741824 (1024.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 2244935680 (2140.9375MB)
   used     = 863166976 (823.18017578125MB)
   free     = 1381768704 (1317.75732421875MB)
   38.44951923076923% used
From Space:
   capacity = 112525312 (107.3125MB)
   used     = 47609824 (45.404266357421875MB)
   free     = 64915488 (61.908233642578125MB)
   42.31032392071928% used
To Space:
   capacity = 114753536 (109.4375MB)
   used     = 0 (0.0MB)
   free     = 114753536 (109.4375MB)
   0.0% used
PS Old Generation
   capacity = 5010817024 (4778.6875MB)
   used     = 4385643424 (4182.475494384766MB)
   free     = 625173600 (596.2120056152344MB)
   87.52351967741699% used
PS Perm Generation
   capacity = 458031104 (436.8125MB)
   used     = 432700088 (412.6549606323242MB)
   free     = 25331016 (24.15753936767578MB)
   94.46958606549131% used

Why are you using ParallelOldGC? Wasn't it worse than the regular paralell one? — Mister Smith, Sep 05 '11 at 14:25
I don't see any evidence of an infinite GC. Can you give us the logs of what happens just before you have to kill it? — Peter Lawrey, Sep 05 '11 at 14:54
I agree with @Peter Lawrey. This looks to me like an infinite loop in your application. If I'm reading it right, the GC log shows that the JVM is spending less that 0.3 seconds every 15 seconds running the garbage collector. — Stephen C, Sep 10 '11 at 06:44
I have seen an infinite GC when the used after the GC is as high or higher than before the GC and it does this repeatedly. What to look for the the after size is the same or larger and there appears to be no time between GCs. — Peter Lawrey, Sep 10 '11 at 06:47
@Peter Lawrey We suspect the same thing as well. Full GC is not able to clear anything which results in another Full GC resulting in freeze. The size before and after full gc remains the same. And all objects are in PS Old Generation. Why it is not clearing objects in PS Old Generation? — Dinesh Varadharajan, Sep 10 '11 at 11:25
I have only seen this situation when you create objects like mad and you fill the old generation space. I beleieve the correct behaviour would be to produce an OutOfMemoryError but for some reason it does not. However you haven't shown us the logs which demonstrate this behaviour you describe so I can't be sure its the same thing. — Peter Lawrey, Sep 10 '11 at 11:32

score 4 · Accepted Answer · answered Sep 05 '11 at 14:32

Can you cancel the ParallelOldGC? It seems to cause memory fragment.

Or you can try to add

-XX:+UseCMSCompactAtFullCollection and -XX:CMSFullGCsBeforeCompaction=0

Also you can add -server It seems always to be used for server side Java application.

Not sure if it can help. Because I cannot try it for you.

score 2 · Answer 2 · answered Sep 05 '11 at 14:24

2

Not sure it helps, but you can use an additional param:

      -XX:ParallelGCThreads=10  //10 threads for GC

to reduce the default number of GC threads.

answered Sep 05 '11 at 14:24

Mister Smith

27,417
21
110
193

2

Should the number of threads be equal to the number of CPU cores in your system? – AngerClown Sep 05 '11 at 14:50
1

@AngerClown These are the threads dedicated to GC. You'll tipically set this number to a lower value than the default, which probably is too high, in order to liberate threads for the actual computation. So you should not use but a fraction of the total number of threads available. – Mister Smith Sep 05 '11 at 15:57

score 0 · Answer 3 · answered Sep 10 '11 at 06:28

0

The problem could be Xmx and Xms both set to the same large value of 7g. Why don't you set the Xms to a lower value? IMHO, this should be corrected.

answered Sep 10 '11 at 06:28

Scorpion

3,938
24
37

We already tried that as well. We set Xms to 1G but we weren't able to arrest this.Thanks for the suggestion. – Dinesh Varadharajan Sep 10 '11 at 09:09

JVM hangs because of infinite GC

3 Answers3