4

How do you configure Apache Cassandra to allow for disaster recovery, to allow for one of two data-centres to fail?

The DataStax documentation talks about using a replication strategy that ensures at least one replication is written to each of your two data-centres. But I don't see how that helps once the disaster has actually happened. If you switch to the remaining data-centre, all your writes will fail because those writes will not be able to replicate to the other data-centre.

I guess you would want your software to operate in two modes: normal mode, for which writes must replicate across both data-centres, and disaster mode, for which they need not. But changing replication strategy does not seem possible.

What I really want is two data-centres that are over provisioned, and during normal operations use the resources of both data-centres, but use the resources of only the one remaining data-centre (with reduced performance) when only one data-centre is functioning.

Raedwald
  • 46,613
  • 43
  • 151
  • 237
  • Quoth the documentation http://wiki.apache.org/cassandra/Operations#Replication: "Replication factor is not really intended to be changed in a live cluster" – Raedwald Nov 30 '12 at 15:32
  • Yet the FAQ does not warn against altering the replication factor of a live cluster: http://wiki.apache.org/cassandra/FAQ#change_replication – Raedwald Nov 30 '12 at 15:40
  • it works, we've done it, but it may increase latency on reads/writes during the repair – tysonjh Nov 30 '12 at 21:34

1 Answers1

10

The trick is to vary the consistency setting given through the API for writes, instead of varying the replication factor. Use the LOCAL_QUORUM setting for writes during a disaster, when only one data-centre is available. During normal operation use EACH_QUORUM to ensure both data-centres have a copy of the data. Reads can use LOCAL_QUORUM all the time.

Here is a summary of the Datastax documentation for multiple data centers and the older but still conceptionally relevant disaster recovery (0.7).

Make a recipe to suite your needs with the two consistencies LOCAL_QUORUM and EACH_QUORUM.

Here, “local” means local to a single data center, while “each” means consistency is strictly maintained at the same level in each data center.

Suppose you have 2 datacenters, one used strictly for disaster recovery then you could set the replication factor to...

3 for the primary write/read center, and two for the failover data center

Now depending how critical it is that your data is actually written to the disaster recovery nodes, you can either use EACH_QUORUM or LOCAL_QUORUM. Assuming you are using a replication placement strategy NetworkTopologyStrategy (NTS),

LOCAL_QUORUM on writes will only delay the client to write locally to the DC1 and asynchronously write to your recovery node(s) in DC2.

EACH_QUORUM will ensure that all data is replicated but will delay writes until both DCs confirm successful operations.

For reads it's likely best to just use LOCAL_QUORUM to avoid inter-data center latency.

There are catches to this approach! If you choose to use EACH_QUORUM on your writes you increase the potential failure points (DC2 is down, DC1-DC2 link is down, DC1 quorum can't be met).

The bonus is once your DC1 goes down, you have a valid DC2 disaster recovery. Also note in the 2nd link it talks about custom snitch settings for routing your IPs properly.

Raedwald
  • 46,613
  • 43
  • 151
  • 237
tysonjh
  • 1,329
  • 1
  • 12
  • 27
  • So, the key is that you can choose the quorum for writes, not just for reads. – Raedwald Dec 01 '12 at 08:43
  • @tysonjh some of the catches you mentioned can be easily overcome with a custom IRetryPolicy which can degrade the level of consistency from EACH_QUORUM to LOCAL_QUORUM. – Scooletz Nov 22 '13 at 09:33
  • If you're willing to degrade to LOCAL_QUORUM, use it in the first place. You don't need EACH_QUORUM to get a copy in each dc. Every replica in every DC will get a copy of the data in both cases. You only need EACH_QUORUM if you want to fail when a DC is down. EACH_QUORUM will also kill latencies unless your DC's are physically next to each other. If you need to tolerate a DC going down, issue queries at LOCAL_QUORUM (or less). If you can't tolerate a dc going down at all, issue at EACH_QUORUM... but that's almost certainly not what you intend. – Mike Lococo Oct 13 '16 at 19:18