1

I am running a 6 node cluster of cassandra 1.2 on an Amazon Web Service VPC with Oracle's 64-bit JVM version 1.7.0_10.

When I'm logged on to one of the nodes (ex. 10.0.12.200) I can run nodetool -h 10.0.12.200 status just fine.

However, if I try to use another ip address in the cluster (10.0.32.153) from that same terminal I get Failed to connect to '10.0.32.153:7199: Connection refused'.

On the 10.0.32.153 node I am trying to connect to I've made the following checks.

  • From 10.0.12.200 I can run telnet 10.0.32.153 7199 and I get a connection, so it doesn't appear to be a security group/firewall issue to port 7199.

  • On 10.0.32.153 if I run netstat -ant|grep 7199 I see

    tcp 0 0 0.0.0.0:7199 0.0.0.0:* LISTEN

    so cassandra does appear to be listening on the port

  • The cassandra-env.sh file on 10.0.32.153 has all of the JVM_OPTS for jmx active

    -Dcom.sun.management.jmxremote.port=7199 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false

The only shot in the dark I've seen while trying to solve this problem while searching the interwebs is to set the following:

JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=10.0.32.153"

But when I do this I don't even get a response. It just hangs.

Any guidance would be greatly appreciated.

jspyeatt
  • 519
  • 2
  • 6
  • 18
  • 1
    Do you have reverse DNS configured? JMX seems happiest when it is, so that IP resolves to hostname that resolves back to the same IP. – Richard Jul 02 '13 at 17:08
  • I don't believe reverse dns is available within the private subnets of a AWS VPC. – jspyeatt Jul 02 '13 at 18:24

1 Answers1

3

The issue did end up being a firewall/security group issue. While it is true that the jmx port 7199 is used, apparently other ports are used randomly for rmi. Cassandra port usage - how are the ports used?

So the solution is to open up the firewalls then configure the cassandra-env.sh to include

JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=<ip>

Community
  • 1
  • 1
jspyeatt
  • 519
  • 2
  • 6
  • 18