1

our cassandra nodes reach nearly 90% load without any requests. In the log we can see this exception. Has anybody at least an idea?

Our software runs without problems in other environments.

INFO [CompactionExecutor:36366] 2017-02-02 12:27:06,708 NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB

DEBUG [SharedPool-Worker-31] 2017-02-02 15:15:43,015 ReadCallback.java:126 - Failed; received 0 of 1 responses WARN [SharedPool-Worker-34] 2017-02-02 15:15:43,015 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-34,5,main]: {} org.xerial.snappy.SnappyError: [NOT_A_DIRECT_BUFFER] destination is not a direct buffer at org.xerial.snappy.Snappy.uncompress(Snappy.java:509) ~[snappy-java-1.1.1.7.jar:na] at org.apache.cassandra.io.compress.SnappyCompressor.uncompress(SnappyCompressor.java:102) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.io.util.CompressedSegmentedFile$Mmap.readChunk(CompressedSegmentedFile.java:323) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:137) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:19) ~[apache-cassandra-3.7.jar:3.7] at com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:2949) ~[caffeine-2.2.6.jar:na] at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$15(BoundedLocalCache.java:1807) ~[caffeine-2.2.6.jar:na] at java.util.concurrent.ConcurrentHashMap.compute(Unknown Source) ~[na:1.8.0_112] at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:1805) ~[caffeine-2.2.6.jar:na] at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:1788) ~[caffeine-2.2.6.jar:na] at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:97) ~[caffeine-2.2.6.jar:na] at com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:66) ~[caffeine-2.2.6.jar:na] at org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:215) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:193) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.io.util.RandomAccessReader.reBufferAt(RandomAccessReader.java:78) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:220) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.io.sstable.format.big.BigTableScanner.seekToCurrentRangeStart(BigTableScanner.java:186) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.io.sstable.format.big.BigTableScanner.access$200(BigTableScanner.java:51) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:285) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:265) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.io.sstable.format.big.BigTableScanner.hasNext(BigTableScanner.java:245) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$4.hasNext(UnfilteredPartitionIterators.java:216) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:289) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:134) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:127) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:123) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:292) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1799) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2466) ~[apache-cassandra-3.7.jar:3.7] at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.8.0_112] at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) [apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.7.jar:3.7] at java.lang.Thread.run(Unknown Source) [na:1.8.0_112] DEBUG [SharedPool-Worker-30] 2017-02-02 15:15:43,819 ReadCallback.java:126 - Timed out; received 1 of 2 responses (including data) DEBUG [SharedPool-Worker-30] 2017-02-02 15:15:52,744 ReadCallback.java:126 - Timed out; received 1 of 2 responses (including data) DEBUG [GossipStage:1] 2017-02-02 15:15:58,789 FailureDetector.java:456 - Ignoring interval time of 2154429836 for /172.26.216.8 DEBUG [GossipStage:1] 2017-02-02 15:15:58,789 FailureDetector.java:456 - Ignoring interval time of 2000371819 for /172.26.216.4 DEBUG [GossipStage:1] 2017-02-02 15:16:00,999 FailureDetector.java:456 - Ignoring interval time of 2209666046 for /172.26.216.8 DEBUG [GossipStage:1] 2017-02-02 15:16:00,999 FailureDetector.java:456 - Ignoring interval time of 2209670711 for /172.26.216.4

itstata
  • 1,058
  • 7
  • 17
  • 1
    https://groups.google.com/forum/#!topic/nosql-databases/Ovz89qTIIDI please share the keyspace and table create statements. Looks like you have misconfigured compression for version of cassandra you are using. – Marko Švaljek Feb 02 '17 at 14:43
  • CREATE KEYSPACE monitor WITH REPLICATION = {'class' : 'SimpleStrategy', 'replication_factor': 3};bloom_filter_fp_chance=0.100000 AND comment='' AND dclocal_read_repair_chance=0.000000 AND gc_grace_seconds=864000 AND read_repair_chance=0.100000 AND compaction={'sstable_size_in_mb': '160', 'class': 'LeveledCompactionStrategy'} AND compression={'sstable_compression': 'SnappyCompressor'}; – itstata Feb 02 '17 at 14:47
  • As i said, the same keyspace and table structure is working on other infrastructure – itstata Feb 02 '17 at 14:49
  • 1
    I understand, but are the cassandra versions the same? if it's something urgent, simply remove the compression part if you don't need it (I think this is staging or something). It also looks that this depends on cassandra version you are using. – Marko Švaljek Feb 02 '17 at 14:50
  • Thanks, i will check this... really crazy – itstata Feb 02 '17 at 14:51
  • also check cassandra.yaml file for property memtable_allocation_type – Marko Švaljek Feb 02 '17 at 14:52
  • After restarting the cluster the problem no longer occurred.untrustworthy... – itstata Feb 06 '17 at 07:15

0 Answers0