2

Cassandra installation with default configs. Just a single node server, 48GB ram, 2 TB hdd. About 80 millions of rows were inserted when it slowed down significantly. New connections are refused with timeout error.

Opssenter throws timeouts as well.

htop shows 1 cassandra process that loads CPU for 100%

iotop shows periodic reads\writes, but really low intensive - so HDD is not a bottleneck

a lot of RAM is still free and nothing has been swapped

nodetool tpstats - didnt work, crushed with "java.net.SocketTimeoutException: Read timed out"

nodetool status - shows that the server works normally (!): UN, load is 122GB, Owns 100%, tokens 256

tail /var/log/cassandra/system.log - nothing informative for me, the last line is

INFO [ScheduledTasks:1] 2014-02-16 04:36:21,824 StatusLogger.java (line 121) system.local

What is happening? How to find the list of currently executed operations? How to find the reason of that behavior? And how to get it back to normal?

Thanks!

P.S. Finally it threw an exception:

ERROR [ReadStage:1550] 2014-02-16 05:22:26,476 CassandraDaemon.java (line 192) Exception in thread Thread[ReadStage:1550,5,main] java.lang.OutOfMemoryError: Java heap space at org.apache.cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:69) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.(CompressedRandomAccessReader.java:76) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:43) at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile.createReader(CompressedPoolingSegmentedFile.java:48) at org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:39) at org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1195) at org.apache.cassandra.db.columniterator.IndexedSliceReader.setToRowStart(IndexedSliceReader.java:108) at org.apache.cassandra.db.columniterator.IndexedSliceReader.(IndexedSliceReader.java:84) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:65) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.(SSTableSliceIterator.java:42) at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:167) at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62) at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:273) at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1560) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379) at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:327) at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65) at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1396) at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1931) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:744)

Spaceman
  • 1,185
  • 4
  • 17
  • 31
  • What version of C* are you using and what is the heap size of your node? – Lyuben Todorov Feb 16 '14 at 10:33
  • @LyubenTodorov Cassandra 2.0.5, heap size is 8GB – Spaceman Feb 16 '14 at 15:06
  • You might want to make more of your system.log available, and if your not in production yet, possibly increase the debug level for extra details. – Jay Feb 17 '14 at 05:23
  • I definitely will. By now, I solved that by restarting cassandra - /etc/init.d/cassandra restart. The data insertion script was created keeping in mind possible issues. But even so, I was surprised a bit that the data insertion continued to work normally. So the 'restart and try again' technique worked well, which sounds odd for any RDBMS-experienced user. – Spaceman Feb 18 '14 at 13:11
  • You should have a heap dump in the directory you started C* from (possibly / or the home directory of the C* user if you are using a deb or rpm package). Do you have a place you can upload that? If not I can make one for you if you email me. – jbellis Feb 19 '14 at 00:00

1 Answers1

0

Are you sure they are actually 80 million rows from Cassandra's perspective? If you are using CQL3 and the first column of your primary key is the same for all rows then they will all be in the same row under the coverts as this is the partition key. This can grind Cassandra to a hault.

Check cfhistograms for the Row Size column and see if you have a massive row.

Christopher Batey
  • 674
  • 4
  • 7
  • yes, there are 80 mlns rows. Due to the algorithm it's impossible to have a lot of columns there (there is a counter adding 1,2,3... to the row key). And I had checked the data before in the opscenter - yes, just 2-3 columns per row. – Spaceman Feb 18 '14 at 13:07
  • Large partitions will cause repair to be unhappy but that's about the extent of the damage they will do. Definitely shouldn't cause it to fall over OOM. – jbellis Feb 19 '14 at 00:02