0

Off late I am getting a quite a lot of exception for a table that is designed using LeveledCompactionStrategy.

com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (timeout while waiting for repair of inconsistent replica) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:69) at com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:291) at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:205) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52)

Right now , I have brought down the intensity a bit by bringing down the gc_grace_seconds to 7200(Did this as it contains rows that has frequently updated contents. The number of clustering keys may be around 1000 but it each one is updated every minute. So , didnt want to run into TomsbtoneThreshold problem) and increased the sstable_size_in_mb to 250 to have it all in one level.

I am trying to figure out what shall be done for this.

1.Should i tweak the read_repair_chance to higher value or speculative_retry to a higher percentile?

2.If I am choosing 1, does LOCAL_QUORUM trigger a read repair for the syncing all the data between its replicas at any cost? So , technically this has no effect? So , in that case , should i lowert the read_repair_chance to a lower value so that the background process takes care of this?

Would be great if someone shall clarify this...

Ananth
  • 971
  • 9
  • 23

1 Answers1

0

I can't give you a scientific answer on this, but by means of my experience (when such a thing happens) it is usually a single node that goes crazy. If you are on a replication level of say 3, and your cluster contains 6 nodes, then I just decommission the node in question, remove it's data completely and then join the cluster again. This rarely happens, but in my case, this is the fastest solution as repairing most of the time never ends, especially if your node is still part of the cluster and has heavy read/write traffic.

pcdoc
  • 96
  • 1
  • 3