13

Elasticsearch is designed to run in cluster mode, all I have to do is to define the relevant node IPs in the cluster via environment variable and as long as network connectivity is available it will connect and join the other nodes to the cluster.

I have 3 nodes, 1 is acting as the docker swarm manager and the other two are workers. I have initialized the manager and joined the worker nodes and everything looks ok from that standpoint.

Now I'm trying to run the elasticsearch container in a way that will allow me to join all nodes to the same elasticsearch cluster, however, I want the nodes to join using their overlay network interface and that means that I need to know the container internal IP addresses at the time of running the docker service create command, how can I do this? Do I have to use something like consul to achieve this?

Some clarifications:

I need to know, at the time of service creation the IP addresses (or DNS names) for all Elasticsearch participants so I could start the cluster correctly. This has to be at the time of creation and not afterwards. Also, as I understand, I can expose ports 9200/9300 for all services and work with the external machine IPs and get it to work, but I would like to use the overlay network to do all these communications (I thought this is what swarm mode is for).

Or Weinberger
  • 7,332
  • 23
  • 71
  • 116

1 Answers1

2

Only a partial solution here. So, when attaching your services to a custom overlay network, you indeed have access to Docker's custom Service discovery feature. I'll detail the networking feature of Docker Swarm mode, before trying to tie it to your problem.

I'll be using the different term of services and tasks, in which a service could be elasticsearch, whereas a task is a single instance of that elasticsearch service.

Docker networking

The idea is that for each services you create, docker assigns a Virtual IP (VIP), and a custom dns alias. You can retrieve this VIP using the docker service inspect myservice command.

But, there is two modes to attach a service to an overlay network dnsrr and VIP. You can select these options using the --endpoint-mode options of docker service create.

The VIP mode (I believe it is the default one, or at least the most used), affects the virtual ip to the service's dns alias. This means that doing an nslookup servicename would return to you a single vip, that behind the scenes, would be linked to one of your container in a round robin fashion. But, there is also a special dns alias that lets you access all of your instances ips (all of your tasks ips) : tasks.myservice.

So in VIP mode you can retrieve all of your tasks ips using a simple nslookup tasks.myservice, where myservice is a service name.

The other mode is dnsrr. This mode simply gets rid of the VIP, and connects the dns alias to the different tasks (=service instances), in a round robin way. This way, you simply have to do a nslookup myservice to retrieve the different service instances ip.

Elasticsearch clustering

Ok so first of all I'm not really familiar with the way elasticsearch lets you cluster. From what I understood from your question, you need when running the elasticsearch binary, give it as a parameter, the adress of all of the other nodes it needs to cluster with.

So what I would do, is to create a custom Elasticsearch image, probably based on the one from the default library, to which I would add a custom Entrypoint that would firstly run a script to retrieve the other tasks ip.

I'd believe that staying in VIP mode is suitable for you, since there is the tasks.myservice dns alias. You'll then need to parse the output to retrieve the tasks ip (and probably remove yours). Then you'll be able to save them in a config file environment variable, or use them as a runtime option for your elasticsearch binary.

Edit: To create a custom overlay network, you will need to use the docker network create command, and use the --network option of docker service create

This is answer is mainly based on the Swarm mode networking documentation

Community
  • 1
  • 1
MagicMicky
  • 3,819
  • 2
  • 37
  • 53
  • 1
    This sounds like a valid solution, thanks for this writeup, I'll try it soon and let you know how it goes. – Or Weinberger Nov 27 '16 at 10:27
  • Now running a service named 'elastic' with 3 replicas, one on each machine using '--endpoint-mode vip', to test things, I'm running bash in one of the elastic containers but can't get anything to resolve. If I do 'ping elastic' or 'ping ' or 'ping node02.elastic' (node02 is one of the machine names) I get 'unable to resolve host' for all of these commands. – Or Weinberger Nov 30 '16 at 12:38
  • @OrWeinberger You might also have to create your own network using `docker network create mynet` and connect your services to this network using the `--network` option of `docker service create` – MagicMicky Nov 30 '16 at 13:42
  • Thanks, that helped a bit, now I can resolve `tasks.elastic` correctly, however I'm now facing a different issue - In my elasticsearch configuration I bind `0.0.0.0` or `_eth0_`, but for some reason, when running an overlay network, I can see that all containers are assigned to two IP addresses **on the same network interface**, one IP is shared across all containers (10.0.0.2) and the other IP seems to be the correct container IP and is unique across containers. When I bind 0.0.0.0 it also binds the 10.0.0.2 address which prevents nodes to connect to each other. – Or Weinberger Nov 30 '16 at 14:22
  • Well, this seems logic, as binding to 0.0.0.0 means binding to all the interfaces. Have you tried using the actual ip address (probably the one from the eth1 interface) ? – MagicMicky Nov 30 '16 at 14:59
  • But why does eth0 contains two IP addresses? – Or Weinberger Nov 30 '16 at 16:34
  • I believe that there is two interfaces (eth0 and eth1) because when connecting to the overlay network, it also connects to the default machine's bridge network. You can see which subnet is used by which docker network using `docker network inspect mynet`. If you do this, you will probably have the overlay network having a subnet of `10.0.0.0/24`, while your machine bridge network `172.17.0.0/16` – MagicMicky Dec 01 '16 at 09:15
  • That's the thing, I have two IP addresses on the same interface, eth0 – Or Weinberger Dec 01 '16 at 09:42
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/129519/discussion-between-magicmicky-and-or-weinberger). – MagicMicky Dec 01 '16 at 10:11