1

We are currently doing some stress tests with ab tool. The single inserts are doing fine in cassandra. However, when it comes to batch inserts, I'm currently dealing with java out of memory error: Java Heap Space.

I have a virtual box machine with Ubuntu server 13.04 installed in it with 2G of memory

I don't know much about internal configuration in cassandra.

I'm just making a batch insert with size 100(100 insert in a BATCH).

After the I see this error, I have no longer cqlsh access, no nodetool access for almost 1 hour.

How can I fix this error in heavy loads ?

NOTE : It doesn't happen on single inserts with a HTTP POST requests.

NOTE : In my column family, I have a key with TimeUUIDType and the column values are int s and varchar s

UPDATE : Test results show that I didn't have anything wrong before 6000 requests. However, when it comes to 7000, the php code throws the following;

Error connecting to 127.0.0.1: Thrift\Exception\TTransportException: TSocket: timed out reading 4 bytes from 127.0.0.1:9160

Morever, cassandra logs the following in heavy loads;

WARN [ScheduledTasks:1] 2013-06-28 03:43:07,931 GCInspector.java (line 142) 
Heap is 0.9231763795560355 full.  You may need to reduce memtable and/or cache sizes.
Cassandra will now flush up to the two largest memtables to free up memory.  Adjust
flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to 
do this automatically
Lyuben Todorov
  • 13,987
  • 5
  • 50
  • 69
aacanakin
  • 2,844
  • 4
  • 25
  • 42

1 Answers1

1

The batch doesn't sound like a large enough dataset to cause the memory problem, so this sounds like a problem with the JVM on the virtual machine. How much memory have you allocated to it?

You can check by starting JConsole (just type jconsole in the terminal / prompt) and viewing the 'Memory' tab, specifically the value under Max:

JVM Memory Stats


You can also get some solid details about what caused the crash thanks to the XX:+HeapDumpOnOutOfMemoryError parameter included in C*'s startup script, its basically a log file storing the stacktrace that caused the memory problem.

Typically the heap size is calculated automatically by the calculate_heap_sizes() function in cassandra-env.sh. You can however override the number that function generated by setting MAX_HEAP_SIZE to a different value. The same variable is used on lines 174 & 175 in cassandra-env.sh JVM_OPTS="$JVM_OPTS -Xmx${MAX_HEAP_SIZE}" for setting the min and max heap size.

Lyuben Todorov
  • 13,987
  • 5
  • 50
  • 69
  • I get the following when I `start/stop` the cassandra service; `xss = -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1001M -Xmx1001M -Xmn100M -XX:+HeapDumpOnOutOfMemoryError -Xss180k` so I guess it's around 1G – aacanakin Jun 28 '13 at 09:38
  • @aacanakin Should be more then enough, what did does the outofmemory stack trace show? – Lyuben Todorov Jun 28 '13 at 09:40
  • It seems that I missed these logs. Currently, I'm dealing with re-initiating the tests but I don't have cqlsh access right now – aacanakin Jun 28 '13 at 10:02
  • Ok I got the server back. I'm starting with keeping track of `system.log`, `output.log`, `error.log` – aacanakin Jun 28 '13 at 10:07
  • @aacanakin Usually this error is logged in a separate file because if the jvm dies because of the error, you cant log said error. [Check this link](http://www.oracle.com/technetwork/java/javase/clopts-139448.html#gbzrr), usually when the error occurs it states the name and location of the file created. – Lyuben Todorov Jun 28 '13 at 10:10
  • It turns out that the server is ok when the batch insert data size is 50. In 100, it goes crazy. btw, it's also tested in amazon cassandra instances. The result is same. – aacanakin Jun 28 '13 at 13:27