4

At some point my application starts to create a lot of temporary arrays, this is expected behaviour, and I want to give a lot of space to Young Generation, so temporary arrays don't get promoted to the Tenured Generation.

JVM options: java -Xmx240g -XX:+UseConcMarkSweepGC -XX:NewRatio=2 -XX:+PrintGCTimeStamps -verbose:gc -XX:+PrintGCDetails

At some point my GC log starts looking like this:

800.020: [GC 800.020: [ParNew: 559514K->257K(629120K), 0.1486790 secs] 95407039K->94847783K(158690816K), 0.1487540 secs] [Times: user=3.34 sys=0.05, real=0.15 secs] 
800.202: [GC 800.202: [ParNew: 559489K->246K(629120K), 0.1665870 secs] 95407015K->94847777K(158690816K), 0.1666610 secs] [Times: user=3.79 sys=0.00, real=0.17 secs] 
800.402: [GC 800.402: [ParNew: 559478K->257K(629120K), 0.1536610 secs] 95407009K->94847788K(158690816K), 0.1537290 secs] [Times: user=3.48 sys=0.02, real=0.15 secs]

I'm very confused by the fact that Young Generation size is 629120K (=629M), while I expect it to be approx. 1/2 (because NewRatio=2) of Tenured Generation size which is 158690816K (=158G). Tenured size generation corresponds with NewRatio and Xms as expected, i.e. it is 2/3 of total heap size.

JVM version: java version "1.7.0_21" Java(TM) SE Runtime Environment (build 1.7.0_21-b11) Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)

Update: I believe that at this point (800 sec of running time) program has peak temporary array usage. If program does not go beyond 629M Young generation size, does it mean I should increase NewRatio? Let's assume I'm planning to give more workload to the program, and I expect that temporary arrays volume to permanent arrays volume ratio will be the same.

I ran the program with NewRatio=8 before, and gc log consists mostly of lines like these:

800.004: [GC 800.004: [ParNew: 186594K->242K(209664K), 0.1059450 secs] 95345881K->95159529K(126655428K), 0.1060110 secs] [Times: user=2.41 sys=0.00, real=0.10 secs] 
800.122: [GC 800.122: [ParNew: 186610K->221K(209664K), 0.1073210 secs] 95345897K->95159522K(126655428K), 0.1073900 secs] [Times: user=2.37 sys=0.07, real=0.11 secs] 
800.240: [GC 800.240: [ParNew: 186589K->221K(209664K), 0.1026210 secs] 95345890K->95159524K(126655428K), 0.1026870 secs] [Times: user=2.34 sys=0.00, real=0.10 secs] 
800.357: [GC 800.357: [ParNew: 186589K->218K(209664K), 0.1043130 secs] 95345892K->95159527K(126655428K), 0.1043810 secs] [Times: user=2.30 sys=0.07, real=0.10 secs]

It makes me think that NewRatio has the impact on Young generation size currently, but it shouldn't because currently Young generation is far below 1/9 of heap size.

Update 2: It's a huge scientific calculation and my solution needs up to 240GB of memory. It is not a memory leak and algorithm is the best I was able to come up with.

fglez
  • 8,422
  • 4
  • 47
  • 78
  • 1
    What is the issue? The young generation will only increase to 1/2*tenured if there is a need. What makes you think that it wont expand more? – Jatin Apr 21 '13 at 10:03
  • Jatin, I updated the question, the issue is that with NewRatio=8 Young generation is smaller at the same point of program execution –  Apr 21 '13 at 10:37
  • CMS does not expand young size dynamically. Changing heap geometry requires full GC in STW pause. – Alexey Ragozin Apr 21 '13 at 14:13
  • 240g is enormously large heap, I suspect overflow is happening somewhere inside of sizing logic. Try to ask on openjdk mailing list hotspot-gc-use@openjdk.java.net, it might be a bug. – Alexey Ragozin Apr 21 '13 at 14:17
  • Alexey Ragozin , I updated the question, my program may need up to 240g of memory. –  Apr 21 '13 at 15:05

2 Answers2

1

Please, could you try running with the AdaptiveSizePolicy disabled, use : -XX-UseAdaptiveSizePolicy?

This should allow you to preset the sizes of each Heap Area and will disable dynamic changes in their sizes at runtime.

Aleš
  • 8,896
  • 8
  • 62
  • 107
0

First of all, check What is the meaning of the -XX:NewRatio and -XX:OldSize JVM flags?

The NewRatio is the ratio of young generation to old generation (e.g. value 2 means max size of old will be twice the max size of young, i.e. young can get up to 1/3 of the heap).

and you will see your young generation size is correct.

I recommend you to check also Are ratios between spaces/generations in the Java Heap constant? and set static sizes from the very beginning using

-Xms240g -Xmx240g -XX:NewSize=120g -XX:MaxNewSize=120g -XX:-UseAdaptiveSizePolicy
Community
  • 1
  • 1
fglez
  • 8,422
  • 4
  • 47
  • 78