I have 8 nodes in my cassandra cluster which has two datacenters with 4 nodes each. I am using cassandra version 2.0.9 in all nodes. For the last 10 days there are some connection disturbances occurred. But it had resolved immediately. But nodes in DC1 can't handshake with nodes in DC2. The message shown as
INFO [HANDSHAKE-/172.21.142.249] 2016-06-08 15:19:23,022 OutboundTcpConnection.java (line 386) Handshaking version with /172.21.142.249
INFO [HANDSHAKE-/172.21.142.249] 2014-06-08 15:19:23,024 OutboundTcpConnection.java (line 406) Cannot handshake version with /172.21.142.249
In nodetool status also shows as down.
If i run nodetool status from DC1 nodes it shows all DC2 nodes are down.
If i run nodetool status from DC2 nodes it shows all DC1 nodes are down.
But both are showing nodetool gossipinfo as normal and some heartbeat value.
After i restart one node from DC2, then i run nodetool status it shows that node and DC1 nodes as UN, others DC2 nodes as DN. Afer i restarted all the nodes from DC2, it's working fine. But can't understand the rootcause of that issue. Can anyone help me?
