Cassandra server throws java.lang.AssertionError: DecoratedKey(...) != DecoratedKey

Question

I'm currently experimenting around with Cassandra.

On the client-side (with Hector) I look up a few keys like this:

ColumnFamilyResult<String, String> result = template.queryColumns(Arrays.asList("key1","key2","key3"));

Most of the time it seems to work. But other times I get a timeout exception on the client:

Caused by: me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
    at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
    at me.prettyprint.cassandra.service.template.ThriftColumnFamilyTemplate$1.execute(ThriftColumnFamilyTemplate.java:100)
    at me.prettyprint.cassandra.service.template.ThriftColumnFamilyTemplate$1.execute(ThriftColumnFamilyTemplate.java:88)
    at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
    at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
    at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
    at me.prettyprint.cassandra.service.template.ThriftColumnFamilyTemplate.sliceInternal(ThriftColumnFamilyTemplate.java:88)
    at me.prettyprint.cassandra.service.template.ThriftColumnFamilyTemplate.doExecuteSlice(ThriftColumnFamilyTemplate.java:46)
    at me.prettyprint.cassandra.service.template.ColumnFamilyTemplate.queryColumns(ColumnFamilyTemplate.java:113)
    at info.gamlor.experiments.Cassandra.readObjectByKey(ComplexCassandra.java:255)

Caused by: TimedOutException()
    at org.apache.cassandra.thrift.Cassandra$get_slice_result.read(Cassandra.java:7772)
    at org.apache.cassandra.thrift.Cassandra$Client.recv_get_slice(Cassandra.java:570)
    at org.apache.cassandra.thrift.Cassandra$Client.get_slice(Cassandra.java:542)
    at me.prettyprint.cassandra.service.template.ThriftColumnFamilyTemplate$1.execute(ThriftColumnFamilyTemplate.java:95)

And on the server this exception shows up:

ERROR 11:33:55,312 Exception in thread Thread[ReadStage:91,5,main]
java.lang.AssertionError: DecoratedKey(4948402862350542345439897754126541659, 6932) != DecoratedKey(132475956107784875457507977471906551877, 726f6f74) in C:\tem
p\cassandra\lib\cassandra\data\CassandraPolepos\ComplexObjects\CassandraPolepos-ComplexObjects-hd-2-Data.db
        at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:58)
        at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
        at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:78)
        at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
        at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:63)
        at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1331)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1193)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1128)
        at org.apache.cassandra.db.Table.getRow(Table.java:378)
        at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
        at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:816)
        at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1250)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

Sometimes the key-values in the DecoratedKey(...) part takes up pages.

Anyone a hint what I'm doing wrong. Or how to investigate this issue.

Thanks.

That looks like a bug that was fixed a really long time ago: https://issues.apache.org/jira/browse/CASSANDRA-866 Can you reproduce with a modern version of Cassandra? (1.0.10 or 1.1.2) — jbellis, Jul 26 '12 at 23:17
Yes, I can reproduce it with Cassandra 1.1.2, on a Windows box. I'll try to come up with a minimalistic example, which creates this issue. — Gamlor, Jul 27 '12 at 09:48
Yes, same here with with newest Cassandra and Hector 1.0-5. ERROR [ReplicateOnWriteStage:537] 2012-08-16 13:10:15,673 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[ReplicateOnWriteStage:537,5,main] — disco crazy, Aug 16 '12 at 13:19
@Gamlor, did you ever solve this? I'm seeing the same problem. — Jason Axelson, Aug 31 '12 at 10:46
@JasonAxelson No...never solved it. For me it only occurred in a test-suite, where I dropped and recreated the keyspace for test runs. The work around for that suite was to create new, with a unique name, namespace for the test runs. That works for me, but not a real solution. Really should write a nice test case for this...damn I'm lazy. — Gamlor, Aug 31 '12 at 19:59

score 8 · Accepted Answer · answered Mar 26 '13 at 23:25

This is either https://issues.apache.org/jira/browse/CASSANDRA-4687 or https://issues.apache.org/jira/browse/CASSANDRA-5202 You can run the command "nodetool invalidatekeycache" on all of the server showing the error. If it keeps coming back you may want to disable key cache. If it is 5202 it should only happen after dropping and recreating a column family using the same name.

score 0 · Answer 2 · answered Nov 19 '12 at 15:48

0

To me it looks like it was a bug.

With the later version (1.1.5) the error disappeared. At least for me.

answered Nov 19 '12 at 15:48

Gamlor

12,978
7
43
70

Sadly, I'm still seeing this in 1.1.8 on linux. Restarting the cassandra service removes the error... (put here for googling purposes) – akaIDIOT Feb 06 '13 at 08:00

score 0 · Answer 3 · answered Feb 15 '13 at 14:55

0

This looks like https://issues.apache.org/jira/browse/CASSANDRA-4687, which is not yet fixed. A workaround is to disable the key cache.

answered Feb 15 '13 at 14:55

Richard

11,050
2
46
33

Cassandra server throws java.lang.AssertionError: DecoratedKey(...) != DecoratedKey

3 Answers3

Linked