Cassandra 2.1 OutOfMemory

Question

I have a Cassandra node with 64G ram (16GB heap, G1GC), and occasionally this happens:

ERROR [Thrift:74] 2017-06-11 13:20:25,710 CassandraDaemon.java:229 - Exception in thread Thread[Thrift:74,5,main]
java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:3236) ~[na:1.8.0_45]
        at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118) ~[na:1.8.0_45]
        at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) ~[na:1.8.0_45]
        at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153) ~[na:1.8.0_45]
        at org.apache.thrift.transport.TFramedTransport.write(TFramedTransport.java:146) ~[libthrift-0.9.2.jar:0.9.2]
        at org.apache.thrift.protocol.TBinaryProtocol.writeBinary(TBinaryProtocol.java:211) ~[libthrift-0.9.2.jar:0.9.2]
        at org.apache.cassandra.thrift.Column$ColumnStandardScheme.write(Column.java:678) ~[apache-cassandra-thrift-2.1.13.jar:2.1.13]
        at org.apache.cassandra.thrift.Column$ColumnStandardScheme.write(Column.java:611) ~[apache-cassandra-thrift-2.1.13.jar:2.1.13]
        at org.apache.cassandra.thrift.Column.write(Column.java:538) ~[apache-cassandra-thrift-2.1.13.jar:2.1.13]
        at org.apache.cassandra.thrift.ColumnOrSuperColumn$ColumnOrSuperColumnStandardScheme.write(ColumnOrSuperColumn.java:673) ~[apache-cassandra-thrift-2.1.13.jar:2.1.13]
        at org.apache.cassandra.thrift.ColumnOrSuperColumn$ColumnOrSuperColumnStandardScheme.write(ColumnOrSuperColumn.java:607) ~[apache-cassandra-thrift-2.1.13.jar:2.1.13]
        at org.apache.cassandra.thrift.ColumnOrSuperColumn.write(ColumnOrSuperColumn.java:517) ~[apache-cassandra-thrift-2.1.13.jar:2.1.13]
        at org.apache.cassandra.thrift.Cassandra$multiget_slice_result$multiget_slice_resultStandardScheme.write(Cassandra.java:14729) ~[apache-cassandra-thrift-2.1.13.jar:2.1.13]
        at org.apache.cassandra.thrift.Cassandra$multiget_slice_result$multiget_slice_resultStandardScheme.write(Cassandra.java:14633) ~[apache-cassandra-thrift-2.1.13.jar:2.1.13]
        at org.apache.cassandra.thrift.Cassandra$multiget_slice_result.write(Cassandra.java:14563) ~[apache-cassandra-thrift-2.1.13.jar:2.1.13]
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:53) ~[libthrift-0.9.2.jar:0.9.2]
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) ~[libthrift-0.9.2.jar:0.9.2]
        at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:205) ~[apache-cassandra-2.1.13.jar:2.1.13]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_45]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_45]

Now it seems like an obvious message to increase my heap, but I am reluctant to do so without knowing why.

It looks like it was trying to do a write, and ran out of space on the heap. Here's a view of jconsole of the node when it crashed:

jconsole_output

We're using kairosdb and that's showing about 5k writes/s for this node (kairsdb.datastore.write_size)

In the meantime I've set CASSANDRA_HEAPDUMP_DIR somewhere it's allowed to write so I can take a look further.

Some config vars:

key_cache_size_in_mb: 256
row_cache_size_in_mb: 0
concurrent_reads: 128
concurrent_writes: 64
concurrent_counter_writes: 64
memtable_allocation_type: offheap_objects
concurrent_compactors: 1
compaction_throughput_mb_per_sec: 48

Any ideas/advice/pointers?

Thanks!

EDIT: Another node died, with this heap output, pointing at JMX?? screenshot

If you already have the heapdump, try loading it into eclipse memory analyzer tool (MAT) (http://www.eclipse.org/mat/downloads.php). It gives a nice pie-chart output which could help visualize what is in the memory. Also what is the number of cores in the machine, concurrent_writes can be a multiple of the no. of cores * 8. And what is the config value on memtable_flush_writers? Probably you should be increasing it if storage is backed by SSDs. — dilsingi, Jun 12 '17 at 00:55
It seems you are using a thrift client. What is the thrift client you have used. And also post how you have created your connection using your thrift client. Are you using any batch statements? — Shoban Sundar, Jun 12 '17 at 06:30
@dilsingi I've just got the heapdump today, looking at the pie chart: `The thread java.lang.Thread @ 0x356384718 Thrift:242 keeps local variables with total size 5,988,684,488 (83.16%) bytes. The memory is accumulated in one instance of "java.util.HashMap$Node[]" loaded by "".` The machine has 8 cores (16 with HT), memtable_flush_writers is currently commented out, so according to the comments will be set to 2? Storage is not backed by SSD, just a large amount of 1TB disks in RAID6 — rilott, Jun 12 '17 at 13:16
@ShobanSundar The thrift client is Kairosdb 1.1.3, I'm not sure whether that uses batch statements but I believe it does — rilott, Jun 12 '17 at 13:19
Wow 6GB of data in single could mean a huge partition from Kairos. How many millions of metrics/tags is used via Kairos? Try "nodetool cfhistogram" on the Kairos data table to check partition size in C*. Only other Cassandra yaml we had to change was thrift_framed_transport_size_in_mb (we set it at 150MB). Check the values from Kairos cluster on kairosdb.datastore.cassandra.write_delay and kairosdb.datastore.cassandra.write_buffer_max_size. The write_buffer controls how much data is being pushed to Cassandra at a stretch. Try reducing it to stabilize the write throughput on Cassandra side. — dilsingi, Jun 12 '17 at 16:06
@dilsingi :) it's a 5 node cluster, it's become extremely popular. It's only used via Kairos, for submission and reads (via grafana). I'm not sure of the exact number, although it will be millions of reads I'm sure! Ah, I was just rolling back my thrift_framed_transport_size_in_mb change from 150 down to 30 because I thought that might be causing it (I assume you looked at [this](https://github.com/kairosdb/kairosdb/issues/354) issue?). kairosdb.datastore.cassandra.write_delay=10000, kairosdb.datastore.cassandra.write_buffer_max_size=2000000. Those two settings weren't set by me. — rilott, Jun 12 '17 at 20:00
We had similar issue dealing with our kairos cluster. But the write buffer seems way too much on kairos side. Try reducing it by magnitude, we have it at 10,000 and write-delay is only 500 — dilsingi, Jun 12 '17 at 20:05
What cluster size do you have and what's your load like? In comparison. Will probably try experimenting with that — rilott, Jun 12 '17 at 20:33

Cassandra 2.1 OutOfMemory

0 Answers0