0

Environment

I have a following environment:

  • Cassandra 2.1.0
  • 5 nodes in one DC, 4 nodes in second DC
  • 2500 writes per seconds
  • minimal reads (usually none, sometimes few)

Problem

After a few weeks of running I cannot read any data from my cluster, because I have ReadTimeoutException like following:

ERROR [Thrift:15] 2015-01-07 14:16:21,124 CustomTThreadPoolServer.java:219 - Error occurred during processing of message.
com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses.

To be precise it is not only problem in my cluster, The second one was described here: Cassandra GC takes 30 seconds and hangs node and I will try to use fix from CASSANDRA-6541 as leshkin suggested

Question

I know that I probably provide not enough information but I don't have an idea for what I should look for. I know that I can query ONE instead of QUORUM, but only QUORUM option provides me reliable data.

Community
  • 1
  • 1
Piotr Stapp
  • 19,392
  • 11
  • 68
  • 116
  • 2
    Use "TRACING ON" before running your query via 'cqlsh' (CQL interactive terminal) to see what's Cassandra busy doing. http://www.datastax.com/documentation/cql/3.0/cql/cql_reference/tracing_r.html – omid Jan 09 '15 at 14:12
  • @omid I do not run my command in CQLSH, but using cassandra driver, and as I know in CQLSH commands doesn't use QUORUM but ONE – Piotr Stapp Jan 09 '15 at 17:56
  • @Garath you set the consistency in cqlsh using the CONSISTENCY command. It defaults to ONE. – Zanson Jan 27 '15 at 20:58
  • @Zanson you are right. I found this option some time ago – Piotr Stapp Jan 27 '15 at 21:18

0 Answers0