Cassandra - Write doesn't fail, but values aren't inserted

Question

I have a cluster of 3 Cassandra 2.0 nodes. My application I wrote a test which tries to write and read some data into/from Cassandra. In general this works fine.

The curiosity is that after I restarted my computer, this test will fail, because after writting I read the same value I´ve write before and there I get null instead of the value, but the was no exception while writing. If I manually truncate the used column family, the test will pass. After that I can execute this test how often I want, it passes again and again. Furthermore it doesn´t matter if there are values in the Cassandra or not. The result is alwalys the same.

If I look at the CLI and the CQL-shell there are two different views:

Does anyone have an ideas what is going wrong? The timestamp in the CLI is updated after re-execution, so it seems to be a read-problem?

A part of my code: For inserts I tried

Insert.Options insert =   QueryBuilder.insertInto(KEYSPACE_NAME,TABLENAME)
                .value(ID, id)
                .value(JAHR, zonedDateTime.getYear())
                .value(MONAT, zonedDateTime.getMonthValue())
                .value(ZEITPUNKT, date)
                .value(WERT, entry.getValue())
                .using(timestamp(System.nanoTime() / 1000));

and

Insert insert = QueryBuilder.insertInto(KEYSPACE_NAME,TABLENAME)
                .value(ID, id)
                .value(JAHR, zonedDateTime.getYear())
                .value(MONAT, zonedDateTime.getMonthValue())
                .value(ZEITPUNKT, date)
                .value(WERT, entry.getValue());

My select looks like

Select.Where select = QueryBuilder.select(WERT)
            .from(KEYSPACE_NAME,TABLENAME)
            .where(eq(ID, id))
            .and(eq(JAHR, zonedDateTime.getYear()))
            .and(eq(MONAT, zonedDateTime.getMonthValue()))
            .and(eq(ZEITPUNKT, Date.from(instant)));

Consistencylevel is QUORUM (for both) and replicationfactor 3

What consistency level are you using for reads and writes? What is the replication factor on your keyspace? — Andy Tolbert, Jul 24 '15 at 14:04
Usually this occurs from using the default ONE consistency level instead of QUORUM. cqlsh shell uses ONE unless explicitly set. If you wait a second or two or try again do you get the correct values? — Chris Lohfink, Jul 24 '15 at 14:33
You say consistency level is QUORUM but I do not see it set on queries built by query builder? — Nenad Bozic, Jul 25 '15 at 09:25
Before executing each statement I do `statement = statement.setConsistencyLevel(ConsistencyLevel.QUORUM);` But even when I wait an hour the test fails. And after truncating the column family the test passes. — Julia, Jul 27 '15 at 06:21

score 5 · Accepted Answer · edited Jun 20 '20 at 09:12

I'd say this seems to be a problem with timestamps since a truncate solves the problem. In Cassandra last write wins and this could be a problem caused by the use of System.nanoTime() since

This method can only be used to measure elapsed time and is not related to any other notion of system or wall-clock time.

...

The values returned by this method become meaningful only when the difference between two such values, obtained within the same instance of a Java virtual machine, is computed.

http://docs.oracle.com/javase/7/docs/api/java/lang/System.html#nanoTime()

This means that the write that occured before the restart could have been performed "in the future" compared to the write after the restart. This would not fail the query, but the written value would simply not be visible due to the fact that there is a "newer" value available.

Do you have a requirement to use sub-millisecond precision for the insert timestamps? If possible I would recommend using System.currentTimeMillis() instead of nanoTime().

http://docs.oracle.com/javase/7/docs/api/java/lang/System.html#currentTimeMillis()

If you have a requirement to use sub-millisecond precision it would be possible to use System.currentTimeMillis() with some kind of atomic counter that ranged between 0-999 and then use that as a timestamp. This would however break if multiple clients insert the same row at the same time.

Thank you. Now I get the microseconds using `long currentTimeMicros = currentTimeMillis * 1000 + nanoTime / 1000 - (nanoTime > 1000000 ? (nanoTime / 1000000) * 1000 : 0);` This worked for me. — Julia, Aug 03 '15 at 09:21
@Marcus Olsson What does this line mean `the write that occured before the restart could have been performed "in the future" compared to the write after the restart`? Does this mean that Cassandra can ignore write if it receives something with future timestamp which is not yet reached in Cassandra? We have seen a similar issue, but we are using `System.currentTimeMillis()` of some other system — Vivek Vardhan, Jul 23 '21 at 06:21

Cassandra - Write doesn't fail, but values aren't inserted

1 Answers1

Linked