0

i have three nodes which should autoconfig into a cluster but for some reason they dont seem to want to cluster up.

The 3 Hyper-V VM Servers are CENTOS 7 installed as compute with just the tool packages installed. the servers have consecutive IP's in the same subnet there is no firewall enabled - they can all ping each other both by DNS and IP. Ports on all servers seem to be open.

[root@manelasticshard01 ~]# netstat -tuplen
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode      PID/Program name
tcp        0      0 10.1.247.246:9200       0.0.0.0:*               LISTEN      993        26721      3913/java
tcp        0      0 10.1.247.246:9300       0.0.0.0:*               LISTEN      993        23402      3913/java
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      0          20731      1549/sshd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      0          19136      2441/master
tcp6       0      0 :::22                   :::*                    LISTEN      0          20733      1549/sshd
udp        0      0 127.0.0.1:323           0.0.0.0:*                           994        16999      913/chronyd
udp6       0      0 ::1:323                 :::*                                994        17000      913/chronyd

same output for all aside for IP addresses

NODE 3

 "cluster_name" : "PACMAN",
  "nodes" : {
    "NlrIawYrR1aq79P6F_Q3zA" : {
      "name" : "node3",
      "transport_address" : "manelasticshard03/10.1.247.244:9300",
      "host" : "10.1.247.244",
      "ip" : "10.1.247.244",
      "version" : "2.2.0",
      "build" : "8ff36d1",
      "http_address" : "manelasticshard03/10.1.247.244:9200",
      "process" : {
        "refresh_interval_in_millis" : 1000,
        "id" : 3968,
        "mlockall" : false
      }
    }
  }
}

NODE 2

"cluster_name" : "PACMAN",
  "nodes" : {
    "9Sv2k73BTFSgxMDRDdpRgA" : {
      "name" : "node2",
      "transport_address" : "manelasticshard02/10.1.247.245:9300",
      "host" : "10.1.247.245",
      "ip" : "10.1.247.245",
      "version" : "2.2.0",
      "build" : "8ff36d1",
      "http_address" : "manelasticshard02/10.1.247.245:9200",
      "process" : {
        "refresh_interval_in_millis" : 1000,
        "id" : 3898,
        "mlockall" : false
      }
    }
  }
}

NODE 1

   "cluster_name" : "PACMAN",
      "nodes" : {
        "RborUe8CS_C7ynX_yFWJ6Q" : {
          "name" : "node1",
          "transport_address" : "manelasticshard01/10.1.247.246:9300",
          "host" : "10.1.247.246",
          "ip" : "10.1.247.246",
          "version" : "2.2.0",
          "build" : "8ff36d1",
          "http_address" : "manelasticshard01/10.1.247.246:9200",
          "process" : {
            "refresh_interval_in_millis" : 1000,
            "id" : 3913,
            "mlockall" : false
          }
        }
      }
    }

my config is currently as follows but i have pretty much been through every setting i can find.

cluster.name: PACMAN
node.name: "node3"
# node.master: false
# node.data: true
network.host: manelasticshard03
# http.port: 9230
discovery.zen.ping.unicast.hosts: [manelasticshard01, manelasticshard02]
discovery.zen.ping.multicast.enabled: false
# discovery.zen.ping.unicast.enabled: true
# discovery.zen.fd.ping_timeout: 30s
# gateway.expected_nodes: 3
# gateway.recover_after_time: 5m
# gateway.recover_after_nodes: 2
# discovery.zen.minimum_master_nodes: 2

Does anyone have any ideas as to why the nodes are not clustering, or of any other tests i can do to find out.

UPDATE - if i set discovery minimum master nodes to two in order to force the cluster to search for a second node i get the following output.

[root@manelasticshard01 ~]# curl http://manelasticshard01:9200/_cluster/health?pretty=true
{
  "error" : {
    "root_cause" : [ {
      "type" : "master_not_discovered_exception",
      "reason" : null
    } ],
    "type" : "master_not_discovered_exception",
    "reason" : null
  },
  "status" : 503
}
Dan
  • 138
  • 2
  • 11
  • have you tried putting IPs in the `discovery.zen.ping.unicast.hosts` config option? Also, you still seem to have `discovery.zen.ping.unicast.enabled` commented out, such that it's not trying to do *any* discovery. – GregL Feb 10 '16 at 17:57
  • hi, i have pretty much tried every combination of settings at this point - my understanding is that i should just be able to enter a cluster name and elasticsearch will do the rest, i have tried multicast false and unicast true explicitly to no avail. – Dan Feb 11 '16 at 08:43
  • all shards report the same [root@manelasticshard01 ~]# curl http://manelasticshard01:9200/_cluster/health?pretty=true { "cluster_name" : "PACMAN", "status" : "green", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "active_primary_shards" : 0, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 } – Dan Feb 11 '16 at 08:44

1 Answers1

1

Even though i had explicitly allowed the port range elasticsearch was using apparently RHEL/CENTOS firewall does not like elasticsearch.

Turning it off completely resolved the issue.

FAIL>

Dan
  • 138
  • 2
  • 11