From last few days i am seeing on some of the nodes in cassandra cluster DSE is getting shutdown due to the error below and i need to kill Java process and restart DSE service.
I have cross checked reads and writes and compactions nothing looks suspicious, but i am seeing full Gc pause on these server just before the issue happening. what can be the reason for this how to debug that ?? how to fine grain why on those particular nodes this is happening when these nodes are serving same requests like rest of the cluster ??
Is this happening because of Full Gc is not getting performed properly, we using G1GC and DSE 4.8.3
ERROR [SharedPool-Worker-25] 2016-12-27 10:14:26,100 JVMStabilityInspector.java:117 - JVM state determined to be unstable. Exiting forcefully due to:java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3181) ~[na:1.8.0_74]
at org.apache.cassandra.db.RangeTombstoneList.copy(RangeTombstoneList.java:112) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.db.DeletionInfo.copy(DeletionInfo.java:104) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.db.AtomicBTreeColumns.addAllWithSizeDelta(AtomicBTreeColumns.java:217) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.db.Memtable.put(Memtable.java:210) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1230) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:359) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_74]
at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
ERROR [SharedPool-Worker-25] 2016-12-27 10:14:28,100 SEPWorker.java:141 - Failed to execute task, unexpected exception killed worker: {}
java.lang.IllegalStateException: Shutdown in progress
at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) ~[na:1.8.0_74]
at java.lang.Runtime.removeShutdownHook(Runtime.java:239) ~[na:1.8.0_74]
at org.apache.cassandra.service.StorageService.removeShutdownHook(StorageService.java:764) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.utils.JVMStabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:119) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.utils.JVMStabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:109) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:68) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:168) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at
org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at
java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
INFO [Thread-6] 2016-12-27 10:14:56,150 DseDaemon.java:420 - DSE shutting down...
