I have two tables
CREATE TABLE IF NOT EXISTS QueueBucket (
queueName text,
bucketId int,
scheduledMinute timestamp,
scheduledTime timestamp,
messageId uuid,
PRIMARY KEY ((queueName, bucketId, scheduledMinute), scheduledTime, messageId)
) WITH compaction = { 'class' : 'LeveledCompactionStrategy' } AND speculative_retry='NONE' ;
CREATE TABLE IF NOT EXISTS InDelivery (
queueName text,
nodeId uuid,
dequeuedMinute timestamp,
messageId uuid,
bucketid int,
dequeuedTime timestamp,
PRIMARY KEY ((queueName, nodeId,bucketId, dequeuedMinute),dequeuedTime, messageId)
);
In the code I perform insert into QueueBucket and delete from indelivery in batch (logged). But during load testing some how the delete from indelivery sometimes doesn't work although insert into QueueBucket works. To confirm this is applied a read from indelivery check immediately after which read the deleted messageId if the messageId still exist prints WARN log .
queueDao.insertMsgInfo(queueName, bucketId, QueueUtils.getMinute(scheduledTime), scheduledTime, messageId);
queuDao.deleteInDelivery(queueName, nodeId, bucketId, bucketMinute, dequeuedTime, messageId);
if(queueServiceMetaDao.hasIndeliveryMessage(inDeliveryPK)) {
log.warn("messageId {} of queue {} bucket {} with node {} dequuedTime {} dequeud minute {} could not get deleted from indelivery.",
messageId,queueName,bucketId, nodeId,QueueUtils.dateToString(dequeuedTime),QueueUtils.dateToString(bucketMinute));
}
in insertMsgInfo and deleteInDelivery methods i am reusing prepared statement .
"INSERT INTO queuebucket (queuename, bucketid , scheduledminute, scheduledtime, messageid ) VALUES ( ? , ? , ? , ? , ? );"
"DELETE FROM indelivery WHERE queuename = ? AND nodeId = ? AND bucketId=? AND dequeuedMinute=? AND dequeuedTime =? AND messageId=? ;"
in hasIndeliveryMessage I am passing the same values wrapping into inDeliveryPrimaryKey as I passed for deleting indelivery data in moveBackToQueueBucket method.
"SELECT messageId FROM indelivery WHERE queuename = ? AND nodeId = ? AND bucketId=? AND dequeuedMinute=? AND dequeuedTime=? AND messageId=? ;"
I am no clue why i see multple warn message "could not get deleted from indelivery." . Please help
I am using cassandra version 2.2.7 it is 6 node cassandra cluster with replication factor 5 and read and write consistency used is QUORUM.
I also gone through link Cassandra - deleted data still there and https://issues.apache.org/jira/browse/CASSANDRA-7810 but this issue is fixed long time ago.in 2.0.11.
Futher update As per Cassandra - Delete not working i also ran nodetool repair but the problem still persist. Should i run compact as well ?
Futher update: I am no more using batch i do simple insert into queuebucket and delete for indelivery and then read the data but still the problem persist
Adding some logs:
2016-07-19 20:39:42,440[http-nio-8014-exec-12]INFO QueueDaoImpl -deleting from indelivery queueName pac01_deferred nodeid 1349d57f-28f5-37d4-9fe1-dfa14dba4a9f bucketId 382 dequeuedMinute 20160719203900000 dequeuedTime 20160719203942310 messageId cc4fb158-f61e-345b-8dcf-3f842fe52d50:
2016-07-19 20:39:42,442[http-nio-8014-exec-12]INFO QueueDaoImpl -Reading from indelivery : queue pac01_deferred nodeId 1349d57f-28f5-37d4-9fe1-dfa14dba4a9f dequeueMinute 20160719203900000 dequeueTime 20160719203942310 messageid cc4fb158-f61e-345b-8dcf-3f842fe52d50 bucketId 382 indeliveryRow Row[cc4fb158-f61e-345b-8dcf-3f842fe52d50]
2016-07-19 20:39:42,442[http-nio-8014-exec-12]WARN QueueImpl -messageId cc4fb158-f61e-345b-8dcf-3f842fe52d50 of queue pac01_deferred bucket 382 with node 1349d57f-28f5-37d4-9fe1-dfa14dba4a9f dequuedTime 20160719203942310 dequeud minute 20160719203900000 could not get deleted from indelivery .
Should i try will cosnsistency ALL ???
