6

Is it possible to put Apache Cassandra into a split-brain scenario where a network partitioned data service continues to run storing and accessing data without access to the full cluster?

If not, what are the techniques used (links to documentation OK) to avoid a split-brain?

If so, (there seems to be tweets about this happening) what are the recovery options in this case?

danilo
  • 103
  • 4
Drew Anderson
  • 408
  • 1
  • 5
  • 13

1 Answers1

8

So, it really depends on a few things.

First off, Cassandra is designed to be partition tolerant, which means it's meant to continue to work under situations like you're describing. For instance, you might have two datacenters defined, and the network connection drops between them. Whether or not your queries return successfully depends on the consistency level that you query your cluster for. If you choose LOCAL_QUORUM, you will get a result back even if the 2 datacenters can't communicate. This is an intended feature of the database.

Each node in the cluster maintains the full topology of the cluster, so when the network connection is resolved, hinted handoff will kick in and the datacenters will resolve their issues. It may be necessary to run nodetool repair to ensure all data is consistent at this point. (it can't hurt)

Jon Haddad
  • 1,372
  • 3
  • 13
  • 20
  • 1
    I'm really curious about what "will resolve their issues" means. Couldn't you have two completely divergent sets of data at that point? Would these sets of data be merged back together? How? (And if so couldn't this lead to some kind of invalid state?) Or would one datacenter completely lose all of its divergent data? – Jay Sullivan Nov 18 '17 at 21:34
  • 3
    Will resolve their issues means if you're within the hint window, all the mutations that were written will be sent over to the machines that missed them in the first place. If there's a conflict, last write wins (each write carries a timestamp). If you didn't resolve the network partition in time, you'll need to run a repair and again, the last write will win. There is no database in existance that can be available during a network partition and maintain consistency. – Jon Haddad Nov 20 '17 at 01:36