2

This is the first time I am working with elasticsearch. The following is my environment/configuration.

  1. I have 3 EC2 Ubuntu 14.04 instances.
  2. I have download and extracted elasticsearch-2.3.0.tar.gz.
  3. I have changed elasticsearch.yml file under elasticsearch/config in each of the instance. I have made the following changes in each of the elasticsearch.yml file.

3.1. EC2 Instance number 1 ( my client node)

cluster.name: MyCluster
node.name: Client
node.master: false
node.data: false
path.data: /home/ubuntu/elasticsearch/data/elasticsearch/nodes/0

discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["aa.aa.aa.aa" , "aa.aa.aaa.aa" , "aaa.a.aa.aa"]

In the above bracket I have provide IP of all my 3 instances.

3.2. EC2 Instance number 2 ( my Master node)

cluster.name: MyCluster
node.name: Master
node.master: true
node.data: true
path.data: /home/ubuntu/elasticsearch/data/elasticsearch/nodes/0

discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["aa.aa.aa.aa" , "aa.aa.aaa.aa" , "aaa.a.aa.aa"]

In the above bracket I have provide IP of all my 3 instances. Note that I have made node.data: true (according to this link)

3.3. EC2 Instance number 3 ( my data node)

cluster.name: MyCluster
node.name: slave_1
node.master: false
node.data: true
path.data: /home/ubuntu/elasticsearch/data/elasticsearch/nodes/0

discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["aa.aa.aa.aa" , "aa.aa.aaa.aa" , "aaa.a.aa.aa"]

In the above bracket I have provide IP of all my 3 instances.

  1. After this configuration I run elasticsearch service on each instance starting from data node then master node and client node in the end.
  2. If I check the node status using curl http://localhost:9200, I am getting json which states that the node is running.

Json response

  1. But when I check the cluster health using curl -XGET 'http://localhost:9200/_cluster/health?pretty=true' I am getting the following error on my client instance.

Master not discovered exception

I hope I am clear with my question and I am going in the right direction.

Thankyou

Community
  • 1
  • 1
Shivkumar Mallesappa
  • 2,875
  • 7
  • 41
  • 68

1 Answers1

2

Elasticsearch 2.0+ defaults to binding all sockets to localhost. This means, by default, nothing outside of that machine can talk to it.

This is explicitly for security purposes and simple development setups. Locally, it works great, but you need to configure it for your environment when it gets more serious. This is also why you can talk to the node via localhost. Basically, you want this when you want more than one node across other machines using the network settings. This works with ES 2.3+:

network:
  bind_host: [ _local_, _global_ ]
  publish_host: _global_

Then other nodes can talk to the public IP, but you still have localhost to simplify working with the node locally (e.g., you--the human--never have to know the IP when SSHed into a box).

As you are in EC2 with Elasticsearch 2.0+, I recommend that you install the cloud-aws plugin (future readers beware: this plugin is being broken into 3 separate plugins in ES 5.x!).

$ bin/plugin install cloud-aws

With that installed, you get a bit more awareness out of your EC2 instances. With this great power, you can add more detail to your ES configurations:

# Guarantee that the plugin is installed
plugin.mandatory: cloud-aws

# Discovery / AWS EC2 Settings
discovery
  type: ec2
  ec2:
    availability_zones: [ "us-east-1a", "us-east-1b" ]
    groups: [ "my_security_group1", "my_security_group2" ]

# The keys here need to be replaced with your keys
cloud:
  aws
    access_key: AKVAIQBF2RECL7FJWGJQ
    secret_key: vExyMThREXeRMm/b/LRzEB8jWwvzQeXgjqMX+6br
    region: us-east-1
  node.auto_attributes: true

# Bind to the network on whatever IP you want to allow connections on.
# You _should_ only want to allow connections from within the network
# so you only need to bind to the private IP
node.host: _ec2:privateIp_

# You can bind to all hosts that are possible to communicate with the
# node but advertise it to other nodes via the private IP (less
# relevant because of the type of discovery used, but not a bad idea).
#node:
#  bind_host: [ _local_, _ec2:privateIp_, _ec2:publicIp_, _ec2:publicDns_ ]
#  publish_host: _ec2:privateIp_

This will allow them to talk by binding the IP address to what is expected. If you want to be able to SSH into those machines and communicate with ES over localhost (you probably do for debugging), then you will want the version commented out with _local_ as a bind_host in that list.

pickypg
  • 22,034
  • 5
  • 72
  • 84
  • Thank you very much sir for sharing all this concepts and points. I have gone through the plugins , I mean just basics but I want to configure it on my own. I just want to understand the configurations and how they work. Could you please suggest me the solutions I can try in my current environment to overcome this problem. And what all other configurations should I understand very well in order to make my cluster work properly. Thankyou – Shivkumar Mallesappa Jul 01 '16 at 04:38
  • You need to set `network.host` so that it binds to the correct IP address for each port. Using the more advanced `network.bind_host` along with `network.publish_host` option is even better, but host will get you going. Note: if you just use host, then you won't be able to use localhost when talking to any node. – pickypg Jul 01 '16 at 04:42
  • Thank you very much sir. I will go through the above configuration options and make my changes accordingly. Once I am done, I will let you know. Thank you – Shivkumar Mallesappa Jul 01 '16 at 04:43
  • Sir finally I am able to setup cluster . I have set value of my `network.host: 0.0.0.0`. **But I have one doubt, before making network.host: 0.0.0.0 , I tried network.host: ["ip_of_ec21", "ip_of_ec2" , "ip_of_ec23"] on all my instances and I faced the following error when I start elasticsearch service. I am getting this error on every instance. Error : ** `nested: ChannelException[Failed to bind to: /xx.xx.xx.x:9400]; nested: BindException[Cannot assign requested address];` And the ip in the above error is ip of my master node instance. I am not able to understand the reason. – Shivkumar Mallesappa Jul 01 '16 at 06:35
  • `network.host` is a _per_ node setting. It should only be set to the relevant information of _that_ node's IP address. For example, `ip_of_ec22` should _only_ be set on the node that corresponds to `ec22`. The node (JVM) is literally trying to bind the socket to that IP address. It cannot do that for an IP that is not associated with the VM it's running in! – pickypg Jul 01 '16 at 15:22
  • 1
    You shouldn't need to put the keys in elasticsearch.yml if you assign a proper IAM role to the instance (one that allows ec2:DescribeInstances). – jpancoast Jul 01 '16 at 15:49
  • Sir I again made changes. node1 : `network.host: ec2-1_public ip`, node2: `network.host: ec2-2_public ip`, node3 : `network.host: ec2-3_public ip`. Now I am getting ` ChannelException[Failed to bind to: /xx.xx.xx.xx:9400]; nested: BindException[Cannot assign requested address]; Likely root cause: java.net.BindException: Cannot assign requested address` error on each of the node. What is the reason for this ? I am extremely sorry if I am pulling this question too long, but I just want to clear the purpose and use of the option. Thank you – Shivkumar Mallesappa Jul 01 '16 at 16:45
  • The fact that it is reporting port 9400 is suspicious. It suggests either a custom transport profile is in use (not shown) or it's trying to assign each port with that IP and failing; then it just stops once it hits 9400. Are you doing more than your configuration above is showing? For each node, you should literally be providing the IP address as long as you don't have that `cloud-aws` plugin. You do need to open up those ports (9200 and 9300) in your AWS firewall for those VMs. – pickypg Jul 01 '16 at 18:06
  • No Sir , I am not providing any port number on any instance. I am just providing elastic ip of each of the instance in the network.host on each individual instance.Thank you – Shivkumar Mallesappa Jul 04 '16 at 06:10
  • Hello Sir , If it is possible for you ,Can we move to a chat room to discuss this issue in details and find some solution or conclusion for it . I can then share my elasticsearch.yml file for all the instances. Thank you – Shivkumar Mallesappa Jul 04 '16 at 08:30
  • Finally it is done. I configured network.host of each instance with the private ip of the instance. and now it is working perfectly. I still did not understood why it is not working with Elastic-ip/public-ip ,but I will definitely find the reason for it. Thank you very much pickypg for sharing your knowledge. – Shivkumar Mallesappa Jul 04 '16 at 09:53