1

I'm using following environment: Debian 9 with Docker CE:

# cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
# docker --version
Docker version 17.09.0-ce, build afdb6d4
#

docker node ls:

# docker node ls | grep elk12
2keku0oj8zhsy6uyvyl4gd4d7     elk12               Down                Active              Reachable
tbwbpkl5qys4wwxbisga3y2oe *   elk12               Ready               Active              Reachable
# docker node inspect elk12
[]
Status: Error response from daemon: node elk12 is ambiguous (2 matches found), Code: 1
#

I cannot use docker node rm elk12, as I according to above output I have 2.

How does one proceed to remove "down" node from list, preferably without affecting working cluster). I don't believe I even have that node anymore (probably some leftovers from long time ago) ...

Please advise.

UPDATE:

# docker node rm 2keku0oj8zhsy6uyvyl4gd4d7
Error response from daemon: rpc error: code = FailedPrecondition desc = node 2keku0oj8zhsy6uyvyl4gd4d7 is a cluster manager and is a member of the raft cluster. It must be demoted to worker before removal
# docker node demote 2keku0oj8zhsy6uyvyl4gd4d7
Manager 2keku0oj8zhsy6uyvyl4gd4d7 demoted in the swarm.
# docker node rm 2keku0oj8zhsy6uyvyl4gd4d7
2keku0oj8zhsy6uyvyl4gd4d7
# docker node ls | grep elk12
tbwbpkl5qys4wwxbisga3y2oe     elk12               Ready               Active              Reachable
#
alexus
  • 13,112
  • 32
  • 117
  • 174

1 Answers1

4

The issue you're having with a duplicated node with the same name but different ID is explained here, probably a node that left the swarm and rejoined it, a manager restart...

If you want to remove the duplicated node and get rid of that ambigous warning, you may use the hash ID and not the name:

docker node rm 2keku0oj8zhsy6uyvyl4gd4d7

If you want to remove the other node named elk12, it seems that the node is a manager so you should proceed with caution as you may run into troubles if there are not enough managers to reach a consensus. If you feel it's safe to remove that manager node because you have enough nodes running as managers in your swarm, you should perform the following:

  1. Demote the manager node (docker node demote). So that node is no longer a manager node and runs as a worker node.
  2. Drain the worker node (docker node drain). The containers running in the elk2 would be moved to other worker nodes in the cluster.
  3. Make the worker node leave the Swarm (docker swarm leave inside the elk2 node)
  4. Remove the node (docker node rm ...)
Miguel A. C.
  • 1,366
  • 11
  • 12
  • I updated my question with attempt to follow your answer. – alexus Nov 21 '17 at 15:44
  • I'm using [this](https://github.com/moby/moby/issues/24106) as reference as I've never seen the same issue as yours. As the node refuses to be deleted because it's a manager node then you can try to demote it as suggested, but the problem is that you must be absolutely sure that when removing that node you still have enough manager nodes as you may not be able to manage the cluster if consensus cannot be reached. I think you may try to open an [issue](https://github.com/moby/moby/issues) explaining the situation and the best way to proceed so you don't break the cluster. – Miguel A. C. Nov 21 '17 at 17:38
  • You can check how many nodes are working as managers with `docker node ls -f "role=manager"` taking into account that "Raft tolerates up to (N-1)/2 failures and requires a majority or quorum of (N/2)+1 members to agree on values proposed to the cluster" where N is the number of managers. Also you may check if you have containers running in the elk `node with docker service ls | grep elk2` those containers should be moved to a different node using the drain option once the manager was demoted to a worker node (`docker node update --availability drain`). – Miguel A. C. Nov 21 '17 at 17:42