5

Over the weekend, we started seeing errors in Cassandra. Essentially, complaining that it couldn't get enough nodes together for SERIAL consistency.

This appeared to be a problem with AWS vpn across regions. So, to simplify, I dropped one the other node (There were only two nodes at the time). I did this by removing the seed from the last remaining node's cassandra.yaml:

seed_provider:
...
- seeds: "single node ip"

I also ran nodetool removenode on the old node (which was showing a dead state).

So the topology is dead simple. Application is a Java app which connects to the Cassandra node via Java API.

Below is the error I am seeing now:

Caused by: com.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency QUORUM (2 required but only 1 alive)
        at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:45)
        at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:34)
        at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:182)
        at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:66)
        ... 21 more

Which is the exact same error as before, except node we have QUORUM where we had SERIAL before.

I just tried setting the replication factor to 1:

ALTER KEYSPACE my_keyspace WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };

and restarted cassandra. No change.

Also just ran nodetool cleanup keyspace_name based on answer below. Also no change.

Just to be clear, the topology is this:

Java App -> Single Cassandra node

Still seeing: Not enough replica available for query at consistency QUORUM (2 required but only 1 alive)

Community
  • 1
  • 1
mtyson
  • 8,196
  • 16
  • 66
  • 106

3 Answers3

9

You mentioned you dropped a node. Given you are using QUORUM and it requires 2 replicas, we can assume you have a replication factor of 2 or 3. How many nodes do you have right now and what is your replication factor? Based on your question I think you are indicating you have 1 left, but I wasn't sure. If you only have 1 node and your RF is 2 or 3, you will never be able to meet quorum consistency.

You could alter your replication factor to 1 to resolve this, i.e.:

ALTER KEYSPACE keyspace_name WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };

If you do this, you should also run nodetool cleanup keyspace_name on each node to get previous replicated data.

Another alternative is to add enough nodes to help you meet a QUORUM consistency level with your replication factor.

Andy Tolbert
  • 11,418
  • 1
  • 30
  • 45
  • 1
    Same was my idea too - based on the error message "Not enough replica available for query at consistency QUORUM (2 required but only 1 alive)". Most likely there was 2 total nodes with the RF 2. I'd say add one more replacement node and run nodetool repair -pr on both immediately. And never have RF = number of nodes – Roman Tumaykin Mar 10 '15 at 04:32
  • I did set replication to 1, and after seeing your answer, ran `cleanup`. Error persists... – mtyson Mar 10 '15 at 04:40
  • That doesn't seem right, with an RF of 1, quorum only requires 1 replica. Are you sure you made the correct adjustment? 'describe keyspace keyspace_name' will show your configured RF. – Andy Tolbert Mar 10 '15 at 04:42
  • 1
    I know, it seems odd. When I describe the keyspace I get: `CREATE KEYSPACE my_keyspace WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'} AND durable_writes = true;` – mtyson Mar 10 '15 at 13:50
  • What does nodetool status output? – Roman Tumaykin Mar 12 '15 at 04:19
  • Just an update on this. I saw in my IRC logs that @mtyson got to the bottom of this. His 'cache' and 'system' keyspaces still had the previous replication factors which was causing this issue. mtyson, did I understand that correctly? – Andy Tolbert Mar 12 '15 at 04:21
  • 1
    @AndyTolbert - That's right, if I remember correctly, it was the `cache` keyspace that was holding on to the Network topology, with a replication factor of 2. Thanks for the reminder. I'll add an answer asap (unless you want to :P) – mtyson Mar 12 '15 at 14:03
  • @mtyson, feel free to add an answer since you found the solution :) – Andy Tolbert Mar 12 '15 at 14:32
  • ALTER KEYSPACE keyspace_name WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 }; resolved my problem, thanks – Aleksey Kiselev Mar 24 '17 at 22:27
0

Change QUORUM to ONE while using single node cluster

Ashok v
  • 77
  • 1
  • 8
0

In addition to all,

Check the cassandra LOGS. "logs/debug..."

if you have too many tombstones at the node side. you should clean tombstones. for example Log output :

*"ReadCommand.java:569 - Read 1000 live rows and 1272327 tombstone cells for query SELECT * FROM "*

Check the log levels also,

KUTAY ZORLU
  • 93
  • 1
  • 9