1

There are four nodes in cluster. assume them are node A, B, C, D. enabled hinted handoff.

1) create a keyspace with RF=2, and create a table.

2) make node B, C down(nodetool stopdaemon),

3) login in node A with cqlsh，set CONSISTENCY ANY, insert into a row(assume the row will be stored in node B and C). The row was successfully inserted even though the node B,C was down, because the consistency level is ANY. the coordinator(node A) wrote hints.

4) make node A down(nodetool stopdaemon), then remove node A(nodetool removenode ${nodeA_hostId})

5) make node B, C come back(nodetool start)

6) login in any node of B, C, D. and execute select statement with partition key of inserted row. But there is no any data that inserted row on step 3.

These steps lead to data(on step 3 was inserted row) lost.

Is there any problem with the steps I performed above?

If yes, How to deal with this situation？

look forward to your reply, thanks.

haihong sun
  • 11
  • 2

2 Answers2

1

CONSISTENCY.ANY will result in data loss in many scenarios. It can be as simple as a polar bear ripping a server off the wall as soon as the write is ACKd to the client (not even applied to a single commitlog yet). This is for writes that are equivelent to being ok with durable_writes=false where latency in client is more important than actually storing the data.

If you want to ensure no data loss, have a RF of at least 3 and use quorum, then any write you get an ack for you can be confident will survive a single node failure. A RF=2 can work with quorum but thats the equivalent of CL.ALL which means any node failure, gc, or hiccup will cause loss of availability.

Important to recognize that hints are not about guaranteed delivery, just possibly reducing the time of convergence when data becomes inconsistent. Repairs within the gc_grace_seconds are still necessary to prevent data loss. If your using weak consistency, durability and low replication you open yourself up for data loss.

Chris Lohfink
  • 16,150
  • 1
  • 29
  • 38
-1

Because removenode not stream the data from the node that will be removed. it tells the cluster I am going out of the cluster and balance the existing cluster. Please refer https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsRemoveNode.html

LetsNoSQL
  • 1,478
  • 1
  • 11
  • 23
  • Thanks for your reply. I got your idea. In normal circumstances, execute 'nodetool decommission' to remove a node. it will stream data and transfer hints, this will not lead to hints data loss. But there will be a scenario where the coordinator has entered an unrecoverable state, and have to execute 'nodetool removenode ${coordinator_hostId}' for it, this will lead to hints data loss. How to deal with this situation？ – haihong sun Dec 14 '18 at 07:18
  • In that case, you need to run repair because in your case node D was online and you had replication factor 2. – LetsNoSQL Dec 14 '18 at 07:27
  • the row will be routed to node B, C(That's mean 2 replications of this record should be stored on node B, C), but the tow nodes(B, C) were down. so node D not have the record, run repair without any effect。Only the hints of coordinator(node A) stored the record. Unfortunately, the node A has entered an unrecoverable state. – haihong sun Dec 14 '18 at 11:42