two nodes in cassandra are not gossiping to each other

Question

Node0:
configuration.yaml file:
cluster_name: 'ServerCluster'
num_tokens: 256
Seeds: ""
listen_address: 10.104.0.15
rpc_address: 10.104.0.15
auto_bootstrap: false
endpoint_snitch: GossipingPropertyFileSnitch

cassandra-env.sh file : JVM_OPTS="$JVM_OPTS -java.rmi.server.hostname=10.104.0.15"
cassandra-rackdc file: dc=DC1rack=RAC1

Node1:
configuration.yaml file:
cluster_name: 'ServerCluster'
num_tokens: 256
Seeds: "10.104.0.15"
listen_address: 10.104.0.20
rpc_address: 10.104.0.20
auto_bootstrap: true
endpoint_snitch: GossipingPropertyFileSnitch

cassandra-env.sh file : JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=10.104.0.20"
cassandra-rackdc file: dc=DC1rack=RAC1

Node0 is up and fine
Node1 error:

INFO  00:03:44 OutboundTcpConnection using coalescing strategy DISABLED
INFO  00:03:45 Handshaking version with /10.104.0.15
ERROR 00:04:16 Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any seeds
    at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1333) ~
[apache-cassandra-2.1.10.jar:2.1.10]
    at org.apache.cassandra.service.StorageService.checkForEndpointCollision
(StorageService.java:540) ~[apache-cassandra-2.1.10.jar:2.1.10]
    at org.apache.cassandra.service.StorageService.prepareToJoin(StorageServ
ice.java:788) ~[apache-cassandra-2.1.10.jar:2.1.10]
    at org.apache.cassandra.service.StorageService.initServer(StorageService
.java:720) ~[apache-cassandra-2.1.10.jar:2.1.10]
    at org.apache.cassandra.service.StorageService.initServer(StorageService
.java:611) ~[apache-cassandra-2.1.10.jar:2.1.10]
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.ja
va:387) [apache-cassandra-2.1.10.jar:2.1.10]
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon
.java:562) [apache-cassandra-2.1.10.jar:2.1.10]
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.jav
a:651) [apache-cassandra-2.1.10.jar:2.1.10]
java.lang.RuntimeException: Unable to gossip with any seeds
    at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1333)
    at org.apache.cassandra.service.StorageService.checkForEndpointCollision
(StorageService.java:540)
    at org.apache.cassandra.service.StorageService.prepareToJoin(StorageServ
ice.java:788)
    at org.apache.cassandra.service.StorageService.initServer(StorageService
.java:720)
    at org.apache.cassandra.service.StorageService.initServer(StorageService
.java:611)
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.ja
va:387)
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon
.java:562)
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.jav
a:651)
Exception encountered during startup: Unable to gossip with any seeds
WARN  00:04:16 No local state or state is in silent shutdown, not announcing     shu

tdown INFO 00:04:16 Waiting for messaging service to quiesce INFO 00:04:16 MessagingService has terminated the accept() thread

Then I changed the seeds(because saw a solution in stackoverflow) for
Node0:
Seeds: "10.104.0.15,10.104.0.20"
Node1:
Seeds: "10.104.0.20"

Node0 is fine
Node1 error:

INFO  00:25:36 Thrift API version: 19.39.0
INFO  00:25:36 CQL supported versions: 2.0.0,3.2.1 (default: 3.2.1)
INFO  00:25:36 Initializing index summary manager with a memory pool size of    101
MB and a resize interval of 60 minutes
INFO  00:25:36 Loading persisted ring state
ERROR 00:25:36 Exception encountered during startup
java.lang.IllegalStateException: Shutdown in progress
    at java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java:
66) ~[na:1.8.0_60]
    at java.lang.Runtime.addShutdownHook(Runtime.java:211) ~[na:1.8.0_60]
    at org.apache.cassandra.service.StorageService.initServer(StorageService
.java:718) ~[apache-cassandra-2.1.10.jar:2.1.10]
    at org.apache.cassandra.service.StorageService.initServer(StorageService
.java:611) ~[apache-cassandra-2.1.10.jar:2.1.10]
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.ja
va:387) [apache-cassandra-2.1.10.jar:2.1.10]
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon
.java:562) [apache-cassandra-2.1.10.jar:2.1.10]
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.jav
a:651) [apache-cassandra-2.1.10.jar:2.1.10]
java.lang.IllegalStateException: Shutdown in progress
    at java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java:
66)
    at java.lang.Runtime.addShutdownHook(Runtime.java:211)
    at org.apache.cassandra.service.StorageService.initServer(StorageService
.java:718)
    at org.apache.cassandra.service.StorageService.initServer(StorageService
.java:611)
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.ja
 va:387)
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon
.java:562)
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.jav
a:651)
Exception encountered during startup: Shutdown in progress

So can you help me to get out this chicken and egg problem? Than you in advance

even I followed the datastax link https://academy.datastax.com/courses/ds210-datastax-enterprise-operations-and-performance-tuning/multiple-data-centers-multiple-0
where he just changes the clustername,listen_address, rpc_address and seeds and when he types nodetool status he was able to see both of them but when i follow the same I see only its status in it respective server

Hi Jon, before opening any ports I thought to stop the windows firewall service now they are communicating so are there any specific ports to be opened? and thank you so much for that question I read everywhere but nobody said a word about port thank you once again — Kate Spade, Oct 16 '15 at 16:22

Adrien Piquerez · Answer 1 · 2015-10-15T16:03:10.193

1

I am not sure this will resolve your issue. But it is a good advice.

All the nodes should have the same seeds and you should have at least one seed by datacenter. Even if the only seed of one node is itself.

If I was you I would try this settings on all nodes :

seeds: "10.104.0.15, 10.104.0.20"

And I do not understand why you did not specify any dc, rack for node 1. Is it intentional ?

edited Oct 15 '15 at 16:03

answered Oct 15 '15 at 15:55

Adrien Piquerez

1,009
10
11

I forgot to mention the rack-dc properties for node1 and Even I tried giving the two ip's in the seeds the both nodes are up and running fine but not gossiping and the below is the error I get
java.lang.IllegalStateException: Shutdown in progress at java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java: 66) ~[na:1.8.0_60] at java.lang.Runtime.addShutdownHook(Runtime.java:211) ~[na:1.8.0_60] at org.apache.cassandra.service.StorageService.initServer(StorageService .java:718) ~[apache-cassandra-2.1.10.jar:2.1.10] – Kate Spade Oct 15 '15 at 16:17

score 1 · Accepted Answer · answered Oct 24 '15 at 00:48

Stop both nodes. Delete the data folder. Change the config to have both nodes as seeds. Start each node again. I suspect there may be some weirdness from the first failed try.

Here's a list of all the ports that need to be open, make sure you can reach them (use telnet to check).

http://docs.datastax.com/en/cassandra/2.0/cassandra/security/secureFireWall_r.html

two nodes in cassandra are not gossiping to each other

2 Answers2