3

I am getting dropped mutation due to GC.MY young Generation GC is taking long time

G1 Young Generation GC in 1804ms. G1 Eden Space: 771751936 -> 0; G1 Old Gen: 5351931920 -> 4995416072; G1 Survivor Space: 83886080 -> 58720256;

MUTATION messages were dropped in last 5000 ms: 814 for internal timeout and 0 for cross node timeout. Mean internal dropped latency: 2874 ms and Mean cross-node dropped latency: 0 ms

In order to avoid this should I decrease the MaxGCPauseMillis option in Cassandra-env.sh.

our default value is 500ms.(JVM_OPTS="$JVM_OPTS -XX:MaxGCPauseMillis=500)

Would tuning this setting will reduce the dropped mutation messages. Do I need to increase or decrease the value.

rishi
  • 41
  • 1
  • 3

1 Answers1

2

1.8s GC may not be the exclusive problem if the mutation took over 5s to be processed. Is there a large amount pending in tpstats? how does cpu load look? You may be writing at a rate that Cassandra cant sustain with your hardware and data model.

You probably want to just try increasing your heap size first. Under 16gb of space is pretty hard to tune with G1. Its easier to help if you include gc logs from the time that took 1.8s, without logs its hard to tell what exactly happened. You can try decreasing gc time target, but it might not help if this long GC was an outlier. You may want to increase region size if taking too long to update remember set. Check for humongous objects and such as well which may be data modeling issue (increasing region size would help). If your getting an evacuation failure from a allocation spike, increase the heap size or set GCReservePercent to 25 instead of 10 to give it more room to handle the spikes.

Chris Lohfink
  • 16,150
  • 1
  • 29
  • 38
  • Thanks,Chris There are no pending tasks – rishi Jul 24 '17 at 21:02
  • Thanks,Chris There are no pending tasks .our heap size is 16gb.Everytime we have the dropped mutation we see that new GEN is take about 2 sec's.so if I decrease parameter value to 250 milliseconds is it of no use.? – rishi Jul 24 '17 at 21:09
  • It depends. If you are running quietly then have a sudden spike in allocations, you may still have problem. This is where you can see outlier GCs (ie most of time its 500ms but once a day when it deserializes a large partition index in a compaction it has a 2s GC). This is where the gc logs are useful, also in determining what to tune to prevent it. – Chris Lohfink Jul 24 '17 at 21:16