Cassandra NoHostAvailableException while there is still alive node

Question

I have two C* 2.0.2 nodes in one DC (with a default configuration in cassandra.yaml) and a keyspace with RF=2. Two clients are connected to this DC with a Datastax Java Driver 1.0.3. Clients read and write data from/to C* with CL=ONE without any errors. But when I shut one node down both clients get a huge amount of exceptions:

com.datastax.driver.core.exceptions.NoHostAvailableException:
All host(s) tried for query failed (no host was tried)

After that bunch of exceptions clients continue to work successfully with another node that still remains alive. What should I do to receive not any NoHostAvailableException because there is at least one alive node at a time and CL=ONE is used?

UPDATE: When I shut one of two nodes I sometimes see the following exception in my app log:

[Reconnection-1] [ERROR] [Control connection] Cannot connect to
any host, scheduling retry

Why both nodes are unavailable if I shut only one down? The second one is still alive at this moment and I can connect to it with cqlsh.

Sorry that my answer didn't help. Have you tried turning on trace logging for com.datastax.driver.core.RequestHandler? It looks like when you turn off the first node, the second one gets thrown out of the pool for some reason. The stack trace of exceptions logged in RequestHandler.logError() method ("Error querying bla-bla-bla") might help to find it out. — Wildfire, Dec 12 '13 at 22:34

score -1 · Answer 1 · answered Dec 12 '13 at 18:49

If you execute a request with CL=ONE, the driver tries to query only a single node. So, if the request to that node fails (or the node is unavailable), exception is thrown immediately. This behaviour is controlled by com.datastax.driver.core.policies.RetryPolicy specified when creating a Cluster.

I'd say that a RetryPolicy that makes a fixed count of retry attempts would fit your needs. Unfortunately, Cassandra Driver 1.0.3 doesn't have it bundled (I'm not sure if later versions do). Still, it might be implemented like this:

public class MyRetryPolicy implements RetryPolicy {

    final int attempts;

    public MyRetryPolicy(int attempts) {
        this.attempts = attempts;
    }

    @Override
    public RetryDecision onReadTimeout(Query query, ConsistencyLevel cl, int requiredResponses, int receivedResponses, boolean dataRetrieved, int nbRetry) {
        return (nbRetry >= attempts) ? RetryDecision.rethrow() : RetryDecision.retry(cl)
    }        

    ... <onWriteTimeout & onUnavailable methods with similar implementation>
}

I'm not sure if MyRetryPolicy(2) will be enough, since I didn't dig into driver internals that deep. Probably, another attempt to send the same request to the same host will be made. You may try MyRetryPolicy(10), it should at least significantly decrease the number of failures.

If some failures will still remain (like 1 of 1000), it may worth to look at com.datastax.driver.core.ConvictionPolicy, find its usages and investigate further.

Custom `RetryPolicy` is the first thing I tried but neither of its callback methods gets called when `NoHostAvailableException` is thrown. — tilex, Dec 12 '13 at 21:16

Cassandra NoHostAvailableException while there is still alive node

1 Answers1