8

I created a docker swarm cluster with 4 nodes out of it 2 is swarm manager (swarm supports multiple manager) I understand if the current manager node goes down then the second manager takes the role of being the swarm manager.

In my case I am firing the rest call to the swarm manager for creating the services with replicas and so on.

At some point if this manager goes down and the second manager becomes manager how do I know ??

Is there any way it gives notified that the particular node is a manager dynamically ??

Please give me the clarification on this ??

abronan
  • 3,309
  • 4
  • 29
  • 38
Vishnu Ranganathan
  • 1,277
  • 21
  • 45

4 Answers4

12

To find out manager nodes, run docker info It lists out current node address as well as manager addresses, among many other info.

The command docker node ls isn't very useful here as it works only from a manager node.

theSushil
  • 460
  • 7
  • 13
  • 1
    If running `docker info` doesn't work for you, it may be because you're running it from a node (like your local machine) that is not connected to the swarm. To make sure you're running `docker info` from a node in the swarm, run `docker-machine ls` and make sure one of the machines has an asterisk in the ACTIVE column. If you don't see any asterisks, pick one of the machine's names from the list and run `docker-machine ssh "docker info"` and that should give you the desired output from a node that is in the swarm. – jeffmjack Jul 07 '18 at 17:28
11

Using the CLI

The quick way to (manually) know if the manager changed is through the CLI:

$ docker node ls

ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
46aqrk4e473hjbt745z53cr3t    node-5    Ready   Active        Reachable
61pi3d91s0w3b90ijw3deeb2q    node-4    Ready   Active        Reachable
a5b2m3oghd48m8eu391pefq5u    node-3    Ready   Active
e7p8btxeu3ioshyuj6lxiv6g0    node-2    Ready   Active
ehkv3bcimagdese79dn78otj5 *  node-1    Ready   Active        Leader

You can find out which Manager is the current Leader under the MANAGER STATUS column.

You can also check individually on each node formatting with the Leader field, for example:

$ docker node inspect <id-node> --format "{{ .ManagerStatus.Leader }}"

true

Using the Docker Remote API

You can also check which one of the manager is the leader programmatically. The docker remote API exposes a Nodes API endpoint that you can use to list nodes.

You can provide a filter as a parameter which takes the key=value form.

You can use that filter to list manager nodes only (with role=manager), then you can parse and filter the JSON output to keep the node whose Leader field is set at true (under ManagerStatus).

Unfortunately there is no leader filter (yet) but I assume that this could be a valid enhancement to propose on the swarmkit issue tracker.

There are no events stream on docker swarm mode yet (this is tracked by this issue on the swarmkit repository). I imagine that in the future, an event will trigger on Leader switch and you will be dynamically notified of the new leader if you are listening to these events (for example if you have a special setup and want to dynamically update entries in consul, nginx or Interlock).

abronan
  • 3,309
  • 4
  • 29
  • 38
3

I used docker-machine to set up my local swarm (running VirtualBox).

If you run docker-machine ls it will already show you asterix (*) in the ACTIVE column for the leader. Alternatively use docker node ls to see a bit more detail - there you'll see whether other managers are reachable and which is the leader explicitly.

You will also need more than two managers for fault tolerance (check here https://docs.docker.com/engine/swarm/admin_guide/). Swarm managers use Raft protocol to get strong consistency. For that they need a majority of manager nodes and majority of 2 is 2, so no fault tolerance yet. Your swarm won't be able to elect new leader if the current fails.

I don't know if you can get notifications on the change of the leader. What you can do for sure is to point to any manager node so if you use REST I can imagine setting up HA proxy that would not use unhealthy nodes and point it to all your managers. Any manager can take service calls and it will forward it to the leading manager for scheduling.

Ondrej Burkert
  • 6,617
  • 2
  • 32
  • 27
  • I agree you have a valid point, in my case i have generated a CA certificates from swarm manager, so the manager will respond to the rest call whom ever is coming with the same CA certificates. in this case the manager cant forward the call to the another manager right ?. I struck here i am trying to figure out a way , thanks for your help that helps me a lot. – Vishnu Ranganathan Oct 26 '16 at 10:47
  • I don't see why the manager wouldn't be able to forward the call. I understood it is an internal Swarm mechanism. If I were you I would play with it and see for myself. If you find an error look for it in their bug tracking system. The current swarm mode is a bit buggy (I've encountered more then 5 already reported bugs). It's hopefully gonna be better with version 1.13. – Ondrej Burkert Oct 27 '16 at 11:05
2

If you use a distributed key/value store, check its log or you could use this command on those two swarm manager, it'll show you the node which is primary. docker -H tcp://< swarm_ip:swarm_port > info

With docker-machine, take a look there, set your shell to connect to the swarm manager and use the docker info command.

hxquangnhat
  • 255
  • 2
  • 6