2

I have a cassandra(version 2.2.8) cluster of total 6 nodes, where 3 nodes are seed nodes. One of the seed node recently went down. I need to replace that dead seed node. My cluster is setup in a way where it cannot survive a loss of more than 1 node. I read this documentation to replace the dead seed node. https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/opsReplaceNode.html

As per the documentation, I am scared to remove the dead seed node from the seed list and do a rolling restart. If for any reason any node doesnt start, Ill lose the data.

How to approach this scenario? Is it ok to not remove the dead seed node from the seed list until the new node is fully up and running? As i already have two working seed nodes already present in the seed list. Please advice.

Eats
  • 477
  • 1
  • 6
  • 13

1 Answers1

3

In short: Yes it is okay to wait with removing the seed node.

Explanation: Seed node configuration do two things:

  1. When adding new nodes. The new node will read the seed configuration to get a first contact point to the Cassandra cluster. After the node has joined the cluster it will save information about all Cassandra nodes in it's system.peers table. For all future starts it will use this information to connect to the cluster, not the seed node configuration.
  2. Cassandra also uses seeds as a way to improve gossip. Basically, seed nodes are more likely to receive gossip messages than normal nodes. This improves the speed at which nodes receive updates about other nodes, such as the status.

Losing a seed node will in your case only impact 2. Since you have two more seed nodes I don't see this as a big issue. I would still do a rolling restart on all nodes as soon as you have updated your seed configuration.

Simon Fontana Oscarsson
  • 2,114
  • 1
  • 17
  • 20
  • Thanks for the clarification. My plan is to first replace the dead node. Once the new node joins the cluster. I`ll update the seed configuration to remove the address of dead node and add the address of the new node in the seed list. Will update this on all nodes and do a rolling restart. – Eats Sep 07 '18 at 15:33
  • One follow up question - When I`m setting up the new node, Should i add the address of the dead seed node in the seed list to match the configuration on other nodes. – Eats Sep 07 '18 at 15:38
  • No, the new node should have the new seed configuration. Just make sure you don't add its own ip in its own seed configuration as seed nodes don't bootstrap. – Simon Fontana Oscarsson Sep 07 '18 at 19:51
  • I was successfully able to replace the node with the guidance you provided. After replacing I updated the seed configuration and did a rolling restart for each node. – Eats Sep 19 '18 at 23:49
  • That's awesome. Thanks for letting me know. – Simon Fontana Oscarsson Sep 20 '18 at 08:04