2

I have a 3 node Cassandra cluster setup (replication set to 2) with Solr installed, each node having RHEL, 32 GB Ram, 1 TB HDD and DSE 4.8.3. There are lots of writes happening on my nodes and also my web application reads from my nodes.

  1. I have observed that all the nodes go down after every 3-4 days. I have to do a restart of every node and then they function quite well till the next 3-4 days and again the same problem repeats. I checked the server logs but they do not show any error even when the server goes down. I am unable to figure out why is this happening.
  2. In my application, sometimes when I connect to the nodes through the C# Cassandra driver, I get the following error

    Cassandra.NoHostAvailableException: None of the hosts tried for query are available (tried: 'node-ip':9042) at Cassandra.Tasks.TaskHelper.WaitToComplete(Task task, Int32 timeout) at Cassandra.Tasks.TaskHelper.WaitToComplete[T](Task``1 task, Int32 timeout) at Cassandra.ControlConnection.Init() at Cassandra.Cluster.Init()`

But when I check the OpsCenter, none of the nodes are down. All nodes status show perfectly fine. Could this be a problem with the driver? Earlier I was using Cassandra C# driver version 2.5.0 installed from nuget, but now I updated even that to version 3.0.3 still this errors persists.

Any help on this would be appreciated. Thanks in advance.

Rao
  • 20,781
  • 11
  • 57
  • 77
Hitesh
  • 3,449
  • 8
  • 39
  • 57
  • Did you look into `/var/log/cassandra/system.log` to know why the nodes are down ? It can be OutOfMemory exception or long GC collection – doanduyhai Apr 07 '16 at 15:47
  • Ya i checked those logs during the node down, they did not have anything marked as errors in the logs, though there are lot of info logs which say `Unexpected exception during request; channel = [....] java.io.IOException: Error while read (....): Connection reset by peer` – Hitesh Apr 07 '16 at 15:58
  • Can you copy paste the complete stack trace somewhere ? And if it's a network connection error, maybe you should look into the system.log file of the origin server from which the error comes – doanduyhai Apr 07 '16 at 21:22

1 Answers1

1

If you haven't done so already, you may want to look at setting your logging levels to default by running: nodetool -h 192.168.XXX.XXX setlogginglevel org.apache.cassandra DEBUG on all your nodes

Your first issue is most likely an OutOfMemory Exception.

For your second issue, the problem is most likely that you have really long GC pauses. Tailing /var/log/cassandra/debug.log or /var/log/cassandra/system.log may give you a hint but typically doesn't reveal the problem unless you are meticulously looking at the timestamps. The best way to troubleshoot this is to ensure you have GC logging enabled in your jvm.options config and then tail your gc logs taking note of the pause times:

grep 'Total time for which application threads were stopped:' /var/log/cassandra/gc.log.1 | less

The Unexpected exception during request; channel = [....] java.io.IOException: Error while read (....): Connection reset by peer error is typically inter-node timeouts. i.e. The coordinator times out waiting for a response from another node and sends a TCP RST packet to close the connection.

Tim Ojo
  • 103
  • 6