I have a ElasticSearch Cluster with 3 Data Master Nodes, one dedicated Client Node & a logstash sending events to Elasticsearch Cluster via the elasticsearch client node.
The Client is not able to connect to the cluster and seeing the below errors in log:-
[2015-10-24 00:18:29,657][DEBUG][action.admin.indices.create] [ESClient] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-10-24 00:18:30,743][DEBUG][action.admin.indices.create] [ESClient] no known master node, scheduling a retry
I have gone through this answer but it is not working for me. My Master-Data elastic search config looks like below:-
cluster.name: elasticsearch
node.name: "ESMasterData1"
node.master: true
node.data: true
index.number_of_shards: 7
index.number_of_replicas: 1
bootstrap.mlockall: true
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["es-master3:9300", "kibana:9300", "es-master2:9300", "es-master1:9300"]
cloud.aws.access_key: AK
cloud.aws.secret_key: J0
ES Client Config looks like below:-
cluster.name: elasticsearch
node.name: "ESClient"
node.master: false
node.data: false
index.number_of_shards: 7
index.number_of_replicas: 1
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["es-master1:9300", "es-master2:9300", "es-master3:9300", "kibana:9300"]
bootstrap.mlockall: true
cloud.aws.access_key: AK
cloud.aws.secret_key: J0
The nodes are having all the standard configuration like JVM Heap set to 30GB & mlockall set to true.
Logstash Output looks like below:
elasticsearch {
index => "j-%{env}-%{app}-%{iver}-%{[@metadata][app_log_time]}"
cluster => "elasticsearch"
host => "kibana"
port => "9300"
protocol => "transport"
}
Telnet is working fine from ES Client Node to ES Master-Data nodes on port 9300. Also all the three ES Master-Data nodes are able to talk to each other. I have also verified TCP & UDP is enabled between the client & data-master machine by using iperf.
I am using Elastic Search Version 1.7.1 on Debian 7
Can some one let me know what is going wrong or how can I debug this?
