6

I have a running elasticsearch cluster and I am trying to connect kibana to this cluster (same node). Currently the page hangs when I try to open the service in my browser using :. . In my kibana pod logs, the last few log messages in the pod are:

{"type":"log","@timestamp":"2017-10-13T17:23:46Z","tags":["listening","info"],"pid":1,"message":"Server running at http://0.0.0.0:5601"}
{"type":"log","@timestamp":"2017-10-13T17:23:46Z","tags":["status","ui settings","error"],"pid":1,"state":"red","message":"Status changed from uninitialized to red - Elasticsearch plugin is red","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-10-13T17:23:49Z","tags":["status","plugin:ml@5.6.3","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - Request Timeout after 3000ms","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}

My kibana.yml file that is mounted into the kibana pod has the following config:

server.name: kibana-logging
server.host: 0.0.0.0 
elasticsearch.url: http://elasticsearch:9300
xpack.security.enabled: false
xpack.monitoring.ui.container.elasticsearch.enabled: true

and my elasticsearch.yml file has the following config settings (I have 3 es pods)

cluster.name: elasticsearch-logs
node.name: ${HOSTNAME}
network.host: 0.0.0.0

bootstrap.memory_lock: false
xpack.security.enabled: false
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.unicast.hosts: ["172.17.0.3:9300", "172.17.0.4:9300", "172.17.0.4:9300"]

I feel like the issue is currently with the network.host field but I'm not sure. What fields am I missing/do I need to modify in order to connect to a kibana pod to elasticsearch if they are in the same cluster/node? Thanks!

ES Service:

apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  labels:
    component: elasticsearch
    role: master
spec:
  type: NodePort
  selector:
    component: elasticsearch
    role: master
  ports:
  - name: http
    port: 9200
    targetPort: 9200
    nodePort: 30303
    protocol: TCP

Kibana Svc

apiVersion: v1
kind: Service
metadata:
  name: kibana
  namespace: default
  labels:
    component: kibana
spec:
  type: NodePort
  selector:
    component: kibana
  ports:
  - port: 80
    targetPort: 5601
    protocol: TCP

EDIT: After changing port to 9200 in kibana.yml here is what i see in the logs at the end when I try and access kibana:

{"type":"log","@timestamp":"2017-10-13T21:36:30Z","tags":["listening","info"],"pid":1,"message":"Server running at http://0.0.0.0:5601"}
{"type":"log","@timestamp":"2017-10-13T21:36:30Z","tags":["status","ui settings","error"],"pid":1,"state":"red","message":"Status changed from uninitialized to red - Elasticsearch plugin is red","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-10-13T21:36:33Z","tags":["status","plugin:ml@5.6.3","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - Request Timeout after 3000ms","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2017-10-13T21:37:02Z","tags":["error","elasticsearch","admin"],"pid":1,"message":"Request error, retrying\nPOST http://elasticsearch:9200/.reporting-*/esqueue/_search?version=true => getaddrinfo ENOTFOUND elasticsearch elasticsearch:9200"}
{"type":"log","@timestamp":"2017-10-13T21:37:32Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2017-10-13T21:37:33Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2017-10-13T21:37:37Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2017-10-13T21:37:38Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2017-10-13T21:37:42Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
appdap1
  • 521
  • 1
  • 6
  • 17
  • Could you post your service definitions? You're trying to connect to "elasticsearch.default:9300" but it is unclear if you have exposed this as a service properly. – vascop Oct 13 '17 at 20:42
  • sure thing, edited my post to add the service definitions, I also changed my file to use elasticsearch:9300 instead of adding the default – appdap1 Oct 13 '17 at 21:28

1 Answers1

3

The issue here is that you exposed Elasticsearch on port 9200 but are trying to connect to port 9300 in your kibana.yml file.

You either need to edit your kibana.yml file to use:

elasticsearch.url: http://elasticsearch:9200

Or change the port in the elasticsearch service to 9300.

vascop
  • 4,972
  • 4
  • 37
  • 50
  • Oh man, I must have been staring at this for too long, i completely missed that. Thanks! However changing the port doesn't seem to have changed the result still hangs – appdap1 Oct 13 '17 at 21:37
  • When I try and access kibana using `minikube service kibana` I see the an error message (See edit) – appdap1 Oct 13 '17 at 21:39
  • Your 3 es pods are being discovered through port 9300. Did you expose that port with a service. Also it is very poor practice to hardcode pod IP's like that. Have you looked into statefulsets? – vascop Oct 13 '17 at 22:38
  • Yeah I'm using statefulsets to create the pods, originally when I tried it using the pod names "es-0:9300, es-1:9300, ..." it didn't seem to work. I don't have any service for that port I used 9300 because on the "important elasticsearch configuration" page in the es docs, it says `Out of the box, without any network configuration, Elasticsearch will bind to the available loopback addresses and will scan ports 9300 to 9305 to try to connect to other nodes running on the same server. ` – appdap1 Oct 13 '17 at 22:53
  • 1
    These are running in different pods though. You need to expose the discovery ports with a service as well. Then you can set it up as `.:9300` – vascop Oct 13 '17 at 22:58
  • I noticed an es yaml file online had two ports for its elasticsearch service, do I have to do something similar, add 9300 as a port under the ES service? Or should this be a seperate file? – appdap1 Oct 13 '17 at 23:01
  • You can add it in the same service or as a different service. Usually it's a different service since the 9300 doesn't need to be exposed externally to the cluster (only intra-es communication) whereas 9200 typically needs to be accessed by outside clients. – vascop Oct 13 '17 at 23:03
  • If its only internal, what type should the service be instead of NodePort? – appdap1 Oct 13 '17 at 23:19
  • 1
    Headless https://kubernetes.io/docs/concepts/services-networking/service/#headless-services – vascop Oct 13 '17 at 23:20
  • Just to confirm, if I create a new service called `elasticsearch-internal` with default type ClusterIP, and expose port 9300 with this service. Then in my elasticsearch.yml file I would do `es-X.elasticsearch-internal:9300`? Do I also need to add 9300 as a container port to my StatefulSet file (have both 9200 and 9300 as containerPorts with different names)? – appdap1 Oct 16 '17 at 14:48