Cassandra WriteTimeOut Exception at ConsistencyLevel.LOCAL_QUORUM

Question

I am facing WriteTimeOutException while writing to Cassandra using CassandraSinkConnector at ConsistencyLevel.LOCAL_QUORUM.

I am using Cassandra 3.3.0 version

So i need help how to handle and fix this exception?

Stack-Trace:-

WorkerSinkTask due to unrecoverable exception.org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:546) org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:302)org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:205)org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:173)org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:170)org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:214)java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Caused by: org.apache.kafka.connect.errors.ConnectException: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency LOCAL_QUORUM (2 replica were required but only 1 acknowledged the write) cassandra.sink.CassandraSinkTask.put(CassandraSinkTask.java:65)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:524)\n\t... 10 more\nCaused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency LOCAL_QUORUM (2 replica were required but only 1 acknowledged the write)\n\tat

Driver Configuration:-

String user="test";
String password="test1234";

cluster = Cluster.builder()
                        .addContactPoints("some host address")
                        .withPort("1234")
                        .withQueryOptions(new QueryOptions().setConsistencyLevel(ConsistencyLevel.LOCAL_QUORUM))
                        .withCredentials(user, password)
                        .withSocketOptions(
                                new SocketOptions()
                                        .setConnectTimeoutMillis(15000)
                                        .setReadTimeoutMillis(0))
                        .build();
                session = cluster.connect();

Hi, welcome to StackOverflow. Please edit the question to include your relevant code, plus what you expect to happen. At present the community does not have sufficient information to help you. Thanks. — MandyShaw, Feb 02 '19 at 21:07
Hi, apologies but I am not a subject matter expert here, only the person who initially triaged the question. — MandyShaw, Feb 14 '19 at 19:31

score 0 · Answer 1 · answered Feb 20 '19 at 05:17

This only happens when you have have multiple nodes in your DataCenter. When you try to write with LOCAL_QUORUM then (Total/2)+1 nodes should be up to write data.

Suppose during write, Out of 3 nodes 2 went DOWN. In this situation you will get above error.

Now there is a possibility once you will see nodes: They will appear UP but due to high load on random times a node goes down & come UP automatically once load become normal on that node.

So you should check the load, Memory Usage or CPU usage on exact same time when you got this error. Ideally you can use Opsceter to monitor these things.

Cassandra WriteTimeOut Exception at ConsistencyLevel.LOCAL_QUORUM

1 Answers1