0

I experience prolonged GC pause (> 10 seconds) in Datastax-enterprise's Cassandra/Solr package. After monitoring for days I found it only happens when GC of CMS Perm Gen happens as shown in the chart. And the long GC happens at EVERY flection point of the chart, when PermGen GC happens. And every time when Perm Gen GC kicks in, there's a long pause that cause client sessions timeout!

https://www.dropbox.com/s/qgdcurprvc1sees/permgen_gc.png

Heap GC is normal and has no pauses, only and always get the long pause in the Non-heap Perm Gen GC, which always happens when the server is at non-peak hours.

![enter image description here][1]

JVM options used by DSE:

-ea -javaagent:/usr/local/dse/resources/cassandra/lib/jamm-0.2.5.jar 
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42
-Xms16384M -Xmx16384M -Xmn5461M -XX:+HeapDumpOnOutOfMemoryError 
-Xss180k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled 
-XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly 
-Djava.net.preferIPv4Stack=true -Dcassandra.load_ring_state=false 
-Dcassandra-foreground=yes -Dsearch-service=true
-Dtomcat.logs=/var/log/dse/tomcat -DName=SI2_DSE
-Ddse.solr.data.dir=/data/solrIndexRamDisk
-Djava.library.path=/usr/local/dse/resources/hadoop/native/Linux-amd64-64/lib

JVM Info

  • Virtual Machine: Java HotSpot(TM) 64-Bit Server VM version 20.12-b01
  • Vendor: Sun Microsystems Inc.
  • JIT compiler: HotSpot 64-Bit Tiered Compilers

Heap Info

  • Current heap size: 10,247,153 kbytes
  • Maximum heap size: 16,218,048 kbytes
  • Committed memory: 16,218,048 kbytes
  • Pending finalization: {0} objects

VM Server Info

  • Operating System: Linux 2.6.32-279.1.1.el6.x86_64
  • Architecture: amd64
  • Number of processors: 32
  • Committed virtual memory: 39,845,596 kbytes
  • Total physical memory: 99,018,824 kbytes
  • Free physical memory: 58,184,572 kbytes
  • Total swap space: 4,194,296 kbytes
  • Free swap space: 4,194,296 kbytes
cheffe
  • 9,345
  • 2
  • 46
  • 57
Alladdin
  • 1

2 Answers2

1

If you can use straight Solr, you could try Heliosearch, which tries to tackle the problem of GC pauses with off-heap data.

http://heliosearch.org/off-heap-filters/

dataguy
  • 31
  • 2
0

Use -XX:+CMSClassUnloadingEnabled, it allows the CMS collector to sweep the permgen during oldgen GC and unload classes that are no longer in use. Link : http://blog.redfin.com/devblog/2012/06/cmsclassunloadingenabled-at-redfin.html#.UwWeO4XqPK0

Use 100 or 200 MB size for perm generation instead of 60 MB.

try it out and share if it solves your problem

Deven
  • 416
  • 3
  • 7
  • It simply delays the bomb :( – Alladdin Feb 25 '14 at 01:54
  • What was the size of perm generation that you used? Previously this 10 sec pause was happening in 5 hours(correct?) what is the frequency now.Would it be helpful if by sizing perm gen you push it off production hours? is you application 24-7 type or is it meant to be restarted daily? – Deven Feb 25 '14 at 07:17
  • MaxPermSize = 85Mb, but the usage max is only 57Mb It is a 24x7 service and we are already doing rolling restart across several servers which do GC at different time. That's quite a operational pain and error prone. – Alladdin Feb 26 '14 at 17:31
  • Try MaxPermSize = 200m -XX:PermSize=200m currently 57/85 ~ 67% utilization of Perm Generation. – Deven Feb 27 '14 at 06:43