7

It looks like keyspace replication is not happening correctly in my Cassandra setup, I need some ideas in troubleshooting this. I have configured multi datacenter cluster but to begin with I have set the keyspace to use SimpleStrategy with RF 3.

The column families exist:

cqlsh:kairosdb> select columnfamily_name from system.schema_columnfamilies where keyspace_name = 'kairosdb';

 columnfamily_name
-------------------
       data_points
     row_key_index
      string_index

(3 rows)

but I am unable to query on them:

cqlsh:kairosdb> select count(*) from data_points limit 100000;
 Traceback (most recent call last):
  File "/usr/bin/cqlsh", line 957, in perform_simple_statement
    rows = self.session.execute(statement, trace=self.tracing_enabled)
  File "/usr/share/cassandra/lib/cassandra-driver-internal-only-2.1.1.post.zip/cassandra-driver-2.1.1.post/cassandra/cluster.py", line 1282, in execute
    result = future.result(timeout)
  File "/usr/share/cassandra/lib/cassandra-driver-internal-only-2.1.1.post.zip/cassandra-driver-2.1.1.post/cassandra/cluster.py", line 2776, in result
    raise self._final_exception
Unavailable: code=1000 [Unavailable exception] message="Cannot achieve consistency level ONE" info={'required_replicas': 1, 'alive_replicas': 0, 'consistency': 1}

This is how I setup the multi datacenter setup:

  1. Use 2 nodes from each DC in the seeds list
  2. Use org.apache.cassandra.locator.GossipingPropertyFileSnitch for the Snitch
  3. Specify different name for DC in the cassandra-rackdc.properties as appropriate for the node

Here's the keyspace is created:

cqlsh:kairosdb> describe keyspace kairosdb;

CREATE KEYSPACE kairosdb WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'}  AND durable_writes = true;

Any ideas what I can do to troubleshoot this?

Alexis Wilke
  • 19,179
  • 10
  • 84
  • 156
vrtx54234
  • 2,196
  • 3
  • 30
  • 53
  • Can you try level one local? Did you have data in the keyspace and alter it to RF 3? – phact Nov 04 '14 at 02:01
  • What is level one local? – vrtx54234 Nov 04 '14 at 05:16
  • In multi DC you can specify local vs each to specify if you want to wait for responses from a remote data center. http://www.datastax.com/documentation/cassandra/2.0/cassandra/dml/dml_config_consistency_c.html – phact Nov 04 '14 at 14:52
  • Can you share details about the process you took to set up multi-dc and configure your RF 3 – phact Nov 04 '14 at 14:53
  • I edited the original post describing how I setup the multi-dc cluster and how the keyspace is created. Is SimpleStrategy for replication compatible with multi-dc setup? Since this was not working, I removed two rings, moved back to SimpleSnitch and this time the keyspace got created correctly. so I am wondering if the keyspace creation is incorrect. – vrtx54234 Nov 04 '14 at 20:14
  • Separately, I am looking for instructions to change a single DC setup to multi DC setup. Can I follow the same instructions as here - http://www.datastax.com/documentation/datastax_enterprise/4.5/datastax_enterprise/deploy/deployMultiDC.html - but without removing any data in the first DC? – vrtx54234 Nov 04 '14 at 20:20
  • The short answer is yes, you can add a new dc without deleting data. When you update your rf data will be streamed accross. – phact Nov 06 '14 at 08:14

2 Answers2

3

In your keyspace creation you have this syntax:

CREATE KEYSPACE kairosdb WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'}  AND durable_writes = true;

However if you wish to use replication across data centers (DC) then you need to use NetworkTopologyStrategy, so for example:

CREATE KEYSPACE kairosdb WITH replication = {'class': 'NetworkTopologyStrategy', 'DC1': 3, 'DC2': 3};

As per the following documentation link, "Use NetworkTopologyStrategy when you have (or plan to have) your cluster deployed across multiple data centers..."

http://www.datastax.com/documentation/cassandra/2.0/cassandra/architecture/architectureDataDistributeReplication_c.html

markc
  • 2,129
  • 16
  • 27
  • Multi datacenter is the next setup, currently it is only one DC. Even between two nodes in the same DC, when using SimpleStrategy, what is the easiest way to see if the data is in sync. – vrtx54234 Jan 09 '15 at 20:02
  • Hi, generally speaking the rule of thumb is to write your data at a higher consistency level than what you read at. People often worry about their data for consistency across nodes, but you need to think about the cluster not the individual nodes. Have a look at this great slidepack from Netflix who are a major adopter of Cassandra for some interesting reading: http://www.slideshare.net/planetcassandra/c-summit-2013-eventual-consistency-hopeful-consistency-by-christos-kalantzis – markc Jan 12 '15 at 11:00
-3

First run below command on any of your nodes:

describe keyspace system_auth;

Check the class of this keyspace,whether it's SimpleStrategy or NetworkTopologyStrategy.

It should be NetworkTopologyStrategy if you have configured Cassandra for multiple DCs.

If the output returns the class as SimpleStrategy, you need to ALTER this keyspace.

COMMAND:

ALTER KEYSPACE "system_auth" WITH REPLICATION = 'class' : 'NetworkTopologyStrategy', 'dc1' : 3, 'dc2' : 2};

After having done this,try creating a new keyspace to check. You will stop getting the Consistency error that you have been facing.

SpicyRaisin
  • 1
  • 1