0

I created a new cluster in Amazon AWS, with 3 instances, following the instructions on the Datastax installation page for Cassandra.

I also set up all of the things in the cassandra.yaml file, as provided by the instructions on Datastax for adding a new node. They all match.

Now, when I run nodetool status, I get 3 nodes in my cluster, which is great! However, 2 of the nodes are down. The ones that are down are the ones that are not the node I am running nodetool status from.

For example: If I run nodetool status from node 1, it shows node0 and node2 down. If I run nodetool status from node0, it shows node1 and node2 down.

Anyone know why this is happening? I set up the security groups as it said to in the documentation.

Thanks

hockeybro
  • 981
  • 1
  • 13
  • 41
  • Are the nodes all in the same security group? Did you configure the broadcast_address to be the public or the private IP in cassandra.yaml? – Karen B Jun 06 '16 at 01:10
  • I configured the broadcast_address to the private IP. And yes, they are in the same security group. – hockeybro Jun 06 '16 at 02:27
  • I changed the broadcast_address to the private IP and it didn't help. What should the IP be in the security group? I set it to "my IP" and it defaulted to an address that is neither my public or private address. – hockeybro Jun 06 '16 at 05:12
  • My cluster is also deployed across one region for now. – hockeybro Jun 06 '16 at 05:19
  • 1
    Have you checked the logs for the down nodes? Probably under `/var/log/cassandra`. Also, on one node, try running `telnet – Karen B Jun 06 '16 at 05:35
  • I am able to connect across port 9042 using telnet and the private IP. – hockeybro Jun 06 '16 at 16:20
  • I changed my security group for port 7000 to open (anywhere) in AWS and it worked. What should the port be since I don't want anyone to be able to access this cluster? Should it be my private port or what? – hockeybro Jun 06 '16 at 16:25
  • It wasn't working when you had port 7000 open to the security group? Are you sure you had that rule there? – Karen B Jun 06 '16 at 16:55
  • Yes, I had that rule there. Once I set port 7000 open to everyone it is working. What IP address should be in port 7000? Right now, I just pressed MyIP in AWS and it filled in a random one. – hockeybro Jun 06 '16 at 17:01
  • Are you using public or private IPs for the seed node setting in cassandra.yaml? – Karen B Jun 06 '16 at 17:05
  • At any rate, if your nodes are trying to communicate on port 7000 using public IPs (which would explain why it only worked after you opened that port to the Internet), something in the config files is forcing that. Double check cassandra.yaml. – Karen B Jun 06 '16 at 17:46
  • according to this it says port 7000 is used for cassandra inter node communication: https://docs.datastax.com/en/cassandra/2.1/cassandra/install/installAMISecurityGroup.html – hockeybro Jun 06 '16 at 18:26
  • Can you tell me what IP address these ports need to be in the security group? – hockeybro Jun 06 '16 at 18:26
  • You shouldn't need to add individual IP addresses to the firewall rule. You need to figure out why the nodes are trying to communicate over public IPs instead of the private IP, which would be somewhere in the Cassandra configs. – Karen B Jun 06 '16 at 18:41
  • According to the datastax documentation, that port is used for inter node communication right? Which is correct? – hockeybro Jun 06 '16 at 18:47
  • Yes, it is. But that doesn't change the fact that you can and should use private IPs for it, so the security group membership would be enough to open it. Double check that you're using 10.x.x.x addresses for the `seeds`, `rpc_address`, `broadcast_address`, and `broadcast_rpc_address` settings in `cassandra.yaml`. – Karen B Jun 06 '16 at 19:14
  • My rpc_address is 0.0.0 and my rpc port is 9160. Is this incorrect? – hockeybro Jun 06 '16 at 22:36
  • Ok, it's fine if rpc_address is 0.0.0.0. What about the other settings I asked about? – Karen B Jun 07 '16 at 02:08
  • They are all set to the public IP address of the node and seed respectively. – hockeybro Jun 07 '16 at 03:56
  • Then that's the problem. You either need to set them to use the private IP so that security group membership is enough, or you need to add each node's public IP to the firewall rule for each port (and maintain that list as you add new nodes). – Karen B Jun 07 '16 at 05:03
  • So you are saying that the broadcast_address, broadcast_rpc_address, and seed should be the private IP? This didn't work. – hockeybro Jun 07 '16 at 19:00
  • I have the configuration in the answer of this post: http://stackoverflow.com/questions/26892330/cassandra-datastax-enterprise-using-amazon-elastic-ip?rq=1 – hockeybro Jun 07 '16 at 19:00

0 Answers0