docker-compose.yml for elasticsearch and kibana

Question

My aim is to get the elasticsearch and kibana images from DockerHub working locally using Docker.

This does the trick and works perfectly...

docker network create mynetwork --driver=bridge

docker run -p 5601:5601 --name kibana -d --network mynetwork kibana 
docker run -p 9200:9200 -p 9300:9300 --name elasticsearch -d --network mynetwork elasticsearch

Today a bird whispered in my ear and said I should learn docker-compose. So I tried to do all of what's above inside a docker-compose.yml.

Here is my attempt.

version: "2.0"
services:
  elasticsearch:
    image: elasticsearch:latest
    ports:
      - "9200:9200"
      - "9300:9300"
    networks:
      - docker_elk
  kibana:
    image: kibana:latest
    ports:
      - "5601:5601"
    networks:
      - docker_elk
networks:
  docker_elk:
    driver: bridge

Unfortunately this does not work. I've been racking my brains as to why I always get the ECONNREFUSED error as shown below when i run docker-compse up.

$ docker-compose up
Starting training_elasticsearch_1
Recreating training_kibana_1
Attaching to training_elasticsearch_1, training_kibana_1
elasticsearch_1  | [2016-11-02 22:39:55,798][WARN ][bootstrap                ] unable to install syscall filter: seccomp unavailable: your kernel is buggy and you should upgrade
elasticsearch_1  | [2016-11-02 22:39:56,036][INFO ][node                     ] [Caliban] version[2.4.1], pid[1], build[c67dc32/2016-09-27T18:57:55Z]
elasticsearch_1  | [2016-11-02 22:39:56,036][INFO ][node                     ] [Caliban] initializing ...
elasticsearch_1  | [2016-11-02 22:39:56,713][INFO ][plugins                  ] [Caliban] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
elasticsearch_1  | [2016-11-02 22:39:56,749][INFO ][env                      ] [Caliban] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/vda2)]], net usable_space [54.8gb], net total_space [59gb], spins? [possibly], types [ext4]
elasticsearch_1  | [2016-11-02 22:39:56,749][INFO ][env                      ] [Caliban] heap size [990.7mb], compressed ordinary object pointers [true]
kibana_1         | {"type":"log","@timestamp":"2016-11-02T22:39:58Z","tags":["status","plugin:kibana@1.0.0","info"],"pid":11,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1         | {"type":"log","@timestamp":"2016-11-02T22:39:58Z","tags":["status","plugin:elasticsearch@1.0.0","info"],"pid":11,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1         | {"type":"log","@timestamp":"2016-11-02T22:39:58Z","tags":["error","elasticsearch"],"pid":11,"message":"Request error, retrying -- connect ECONNREFUSED 172.20.0.2:9200"}
kibana_1         | {"type":"log","@timestamp":"2016-11-02T22:39:58Z","tags":["status","plugin:kbn_vislib_vis_types@1.0.0","info"],"pid":11,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1         | {"type":"log","@timestamp":"2016-11-02T22:39:58Z","tags":["warning","elasticsearch"],"pid":11,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1         | {"type":"log","@timestamp":"2016-11-02T22:39:58Z","tags":["warning","elasticsearch"],"pid":11,"message":"No living connections"}
kibana_1         | {"type":"log","@timestamp":"2016-11-02T22:39:58Z","tags":["status","plugin:elasticsearch@1.0.0","error"],"pid":11,"state":"red","message":"Status changed from yellow to red - Unable to connect to Elasticsearch at http://elasticsearch:9200.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
kibana_1         | {"type":"log","@timestamp":"2016-11-02T22:39:58Z","tags":["status","plugin:markdown_vis@1.0.0","info"],"pid":11,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1         | {"type":"log","@timestamp":"2016-11-02T22:39:58Z","tags":["status","plugin:metric_vis@1.0.0","info"],"pid":11,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1         | {"type":"log","@timestamp":"2016-11-02T22:39:58Z","tags":["status","plugin:spyModes@1.0.0","info"],"pid":11,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1         | {"type":"log","@timestamp":"2016-11-02T22:39:58Z","tags":["status","plugin:statusPage@1.0.0","info"],"pid":11,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1         | {"type":"log","@timestamp":"2016-11-02T22:39:58Z","tags":["status","plugin:table_vis@1.0.0","info"],"pid":11,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1         | {"type":"log","@timestamp":"2016-11-02T22:39:58Z","tags":["listening","info"],"pid":11,"message":"Server running at http://0.0.0.0:5601"}
elasticsearch_1  | [2016-11-02 22:39:58,515][INFO ][node                     ] [Caliban] initialized
elasticsearch_1  | [2016-11-02 22:39:58,515][INFO ][node                     ] [Caliban] starting ...
elasticsearch_1  | [2016-11-02 22:39:58,587][INFO ][transport                ] [Caliban] publish_address {172.20.0.2:9300}, bound_addresses {[::]:9300}
elasticsearch_1  | [2016-11-02 22:39:58,594][INFO ][discovery                ] [Caliban] elasticsearch/1Cf9qz7CSCqHBEEuwG7PQw
kibana_1         | {"type":"log","@timestamp":"2016-11-02T22:40:00Z","tags":["warning","elasticsearch"],"pid":11,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1         | {"type":"log","@timestamp":"2016-11-02T22:40:00Z","tags":["warning","elasticsearch"],"pid":11,"message":"No living connections"}
elasticsearch_1  | [2016-11-02 22:40:01,650][INFO ][cluster.service          ] [Caliban] new_master {Caliban}{1Cf9qz7CSCqHBEEuwG7PQw}{172.20.0.2}{172.20.0.2:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
elasticsearch_1  | [2016-11-02 22:40:01,661][INFO ][http                     ] [Caliban] publish_address {172.20.0.2:9200}, bound_addresses {[::]:9200}
elasticsearch_1  | [2016-11-02 22:40:01,661][INFO ][node                     ] [Caliban] started
elasticsearch_1  | [2016-11-02 22:40:01,798][INFO ][gateway                  ] [Caliban] recovered [1] indices into cluster_state
elasticsearch_1  | [2016-11-02 22:40:02,149][INFO ][cluster.routing.allocation] [Caliban] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana][0]] ...]).
kibana_1         | {"type":"log","@timestamp":"2016-11-02T22:40:03Z","tags":["status","plugin:elasticsearch@1.0.0","info"],"pid":11,"state":"green","message":"Status changed from red to green - Kibana index ready","prevState":"red","prevMsg":"Unable to connect to Elasticsearch at http://elasticsearch:9200."}
^CGracefully stopping... (press Ctrl+C again to force)
Stopping training_kibana_1 ... done
Stopping training_elasticsearch_1 ... done

Can someone please help me with why?

thanks

Have you tried re-starting Kibana? Maybe elasticsearch isn't fully up when Kibana tries to connect to it? — Roman, Nov 02 '16 at 06:44
Hi @R0MANARMY. I think you are right. Can you check the extra information I've put in the logs to confirm. Your comment has prompted me to add some extra log information to illustrate more about what's happening. Is the fix to add a start dependency or sequence in docker-compose files? How would you do that. — Richie, Nov 02 '16 at 22:48

score 17 · Accepted Answer · answered Jul 18 '17 at 15:38

To add the hard dependency on elasticsearch for kibana, you need the depends_on variable to be set as shown below. Also, to add to @Phil McMillan's answer, you can set the elasticsearch_url variable in kibana, without static addressing using Docker's inbuilt DNS mechanism.

version: '2.1'
services:
     elasticsearch:
       image: docker.elastic.co/elasticsearch/elasticsearch:5.4.3
       container_name: elasticsearch
       networks:
           docker-elk:

     kibana:
       image: docker.elastic.co/kibana/kibana:5.4.3
       container_name: kibana
       environment:
          - "ELASTICSEARCH_URL=http://elasticsearch:9200"
       networks:
          - docker-elk
       depends_on:
          - elasticsearch

networks:
  docker-elk:
    driver: bridge

Note the environment variable ELASTICSEARCH_URL=http://elasticsearch:9200 just uses has the container name (elasticsearch) which the Docker DNS server is able to resolve.

This answer is mega out of date now – Piotr Kula Nov 02 '22 at 10:21 — Piotr Kula, Nov 02 '22 at 10:21

Tuan · Answer 2 · 2016-11-03T02:50:47.793

16

You need to include the links.

version: "2.0"
services:
  elasticsearch:
    image: elasticsearch:latest
    ports:
      - "9200:9200"
      - "9300:9300"
    networks:
      - docker_elk
  kibana:
    image: kibana:latest
    ports:
      - "5601:5601"
    links:
      - elasticsearch
    networks:
      - docker_elk
networks:
  docker_elk:
    driver: bridge

UPDATED

When using the image elasticsearch:latest, it's Elasticsearch 5.0 and requires us to increase our Docker host virtual memory.

Before running the docker-compose, please make sure to run this command on your Docker host.

Linux:

su root
sysctl -w vm.max_map_count=262144

Windows (boot2docker)

docker-machine ssh default
sudo sysctl -w vm.max_map_count=262144

If you don't want to change your Docker host, just use the Elasticsearch 2.x image at elasticsearch:2

edited Nov 03 '16 at 02:50

answered Nov 02 '16 at 07:07

Tuan

2,303
2
25
37

Hi, I tried that. But I still get the same error. I'm thinking it might more be to do with elasticsearch not being fully initialised before kibana comes up. As per the comment by @R0MANARMY. I've edited my original post to show some more logs which I think might illustrate the problem better. – Richie Nov 02 '16 at 22:43
Hi @Tuan. I thought the services running on the same network was enough to get them talking. In fact I thought networks was the successor to links? But are you saying you need to use both together? – Richie Nov 02 '16 at 22:51
@Richie, the best way to check is going into the kibana container. Starting with the ```docker-compose -d up```. Access the kibana by ```docker exec -it training_kibana_1 /bin/bash```, then ```curl http://elasticsearch:9200/```. – Tuan Nov 03 '16 at 02:35
@Richie, do you get this error in Elasticsearch ```max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]``` ? If yes, we should increase the vm.max_map_count in our Docker host. I updated my answer. – Tuan Nov 03 '16 at 02:41
no I don't get that error. Nethertheless I will try to downgrade my versions of kibana and elasticsearch to see if that makes a difference. – Richie Nov 03 '16 at 06:42
1

@Richie, I re-reviewed your logs. It's ES ```2.4.1```! I believe you had the old images of ES and Kibana. You might need to re-pull them before running docker-compose, like ```docker pull elasticsearch``` and ```docker pull kibana``` . – Tuan Nov 03 '16 at 07:38
To get it to work I had to add the following environment variables `environment:` `- "discovery.type=single-node"` `-"ES_JAVA_OPTS=-Xms512m -Xmx512m"` – jbejar Aug 14 '19 at 22:27
this answer is mega out of date now – Piotr Kula Nov 02 '22 at 10:22
@PiotrKula, yes, it was 2016. – Tuan Nov 03 '22 at 05:49

score 3 · Answer 3 · edited Jan 29 '22 at 10:20

3

This works for me docker-compose.yml

    version: '3'
    services:
      elasticsearch:
        image: docker.elastic.co/elasticsearch/elasticsearch:7.6.2
        environment:
          - discovery.type=single-node
        ports:
          - 9200:9200
      kibana:
        image: docker.elastic.co/kibana/kibana:7.6.2
        ports:
          - 5601:5601

edited Jan 29 '22 at 10:20

NanoNova

829
10
19

answered Apr 12 '20 at 10:15

Zahirul Haque

1,599
17
22

1

Do you have a compose file for ES 8.2+ - They use service accounts for kibana now I can't work out how to configure that in the compose section – Piotr Kula Nov 02 '22 at 10:22

score 3 · Answer 4 · answered Jan 25 '22 at 07:24

I'm using docker-compose v3 format according to this post:

version: '3'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.10.2
    container_name: elasticsearch
    environment:
      - node.name=es-node
      - cluster.name=es-cluster
      - discovery.type=single-node
    ports:
      - 9200:9200
      - 9300:9300
    volumes:
      - local-es:/usr/share/elasticsearch/data
    networks:
      - es-net

  kibana:
    image: docker.elastic.co/kibana/kibana:7.10.2
    container_name: kibana
    environment:
      - "ELASTICSEARCH_URL=http://elasticsearch:9200"
    ports:
      - 5601:5601
    networks:
      - es-net
    depends_on:
      - elasticsearch
    restart: "unless-stopped"

networks:
  es-net:

volumes:
  local-es:

Do you have a compose file that works with 8.2+ - They changed the way kibana authenticates using a service account now. I don't know how to update the compose file to create that account — Piotr Kula, Nov 02 '22 at 10:23
I'd recommend going off the definitive source: https://www.elastic.co/guide/en/elasticsearch/reference/8.5/docker.html#docker-file — hoffmanc, Nov 23 '22 at 17:46

shilovk · Answer 5 · 2022-12-08T19:52:57.657

File docker-compose.yml:

version: '3'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.5.2
    container_name: elasticsearch
    environment:
      - node.name=es-node
      - cluster.name=es-cluster
      - discovery.type=single-node
      - xpack.security.enabled=false
    ports:
      - 9200:9200
      - 9300:9300
    volumes:
      - ./docker-data/elasticsearch/data:/usr/share/elasticsearch/data
    networks:
      - elastic
  kibana:
    image: docker.elastic.co/kibana/kibana:8.5.2
    container_name: kibana
    ports:
      - 5601:5601
    networks:
      - elastic
    depends_on:
      - elasticsearch
    restart: 'unless-stopped'

networks:
  elastic:

Official documentation:

Install Kibana with Docker

Phil McMillan · Answer 6 · 2017-02-26T11:58:37.127

I have this working. No links are needed and it doesn't have anything to do with elasticsearch starting before kibana. The issue is that when running under compose, a new bridged network is defined with its own set of IPs. Kibana needs to communicate with the cluster over this bridged network - "localhost" is not available anymore for the connectivity.

You need to do a couple of things:

You need to set a couple of values in kibana.yml or under the environment: section of kibana in the compose file):

a. elasticsearch.url in kibana.yml (or ELASTICSEARCH_URL under the environment: section of kibana in the compose file) must be set to the specific IP of the cluster and port 9200 - localhost will not work, as it does when you run outside of compose.

elasticsearch.url: "http://172.16.238.10:9200"

b. You also need to set server.host (SERVER_HOST) to the bridged IP of the Kibana container.

server.host: "172.16.238.12"

Note: you still access the kibana UI from with http://127.0.0.1:5601 and you still need those "ports" commands!

You need to set an "ipam" configuration under your bridged network and assign elasticsearch and kibana static ips so that kibana can access it via its configuration above.

Something like this should suffice:

elasticsearch:
  networks:
    esnet:
      ipv4_address: 172.16.238.10
kibana:
  networks:
    esnet:
      ipv4_address: 172.16.238.12
networks:
  esnet:
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 172.16.238.0/24

Don't forget to use one of the documented methods to set Kibana configuration - ELASTICSEARCH_URL is required to be set!

I have a docker compose file that creates two elasticsearch nodes and a kibana instance all running on the same bridged network. It is possible.

this answer is mega out of date now – Piotr Kula Nov 02 '22 at 10:22 — Piotr Kula, Nov 02 '22 at 10:22

docker-compose.yml for elasticsearch and kibana

6 Answers6