We are running a cassandra cluster with a local data-center of 3 nodes and (I believe) a replication factor of 3. A remote data-center exist, but isn't used in our queries.
We use Java libraries, more precisely :
- org.apache.cassandra:cassandra-all coupled with com.datastax.cassandra:cassandra-driver-core both v3.1.1 for database interaction
- info.archinnov:achilles-core v5.1.3 for code source annotation processing - this is where the LOCAL_QUORUM consistency level is specified
Recently, we've faced issues related to read after write consistency for new row insert.
A row is inserted using LOCAL_QUORUM consistency. Few milliseconds after the successful insert, the row select by partition key with LOCAL_QUORUM consistency is sometimes - rarely - unsuccessful. No row can be found.
From my naive understanding of LOCAL_QUORUM consistency, if the row is written then read using LOCAL_QUORUM consistency (with same data-center), it should "always" be found and read. But it isn't what seems to be happening.
After reading the Cassandra documentation, I'm wondering if clock synchronization issues could explain the issue. Example : the node without the row could have a clock that is in advance and thus "no row" might be considered as the most updated value.
Is it indeed possible that node clock synchronization issues could explain read after write inconsistency for row insert (despite LOCAL_QUORUM and single local data center) ? If not, what could explain such a consistency issue ?
