I have a Docker swarm with three nodes to which I added two more nodes. All nodes have the same OS (Ubuntu) and the same Docker version (19.03.6). Everything seems to be alright, all nodes are up, the networks span all nodes, when I scale a service to 4 replicas one of them is put on one of the new nodes.
However, while the service responds when using the IP adresses of the first three nodes, it cannot be accessed on the new ones.
Some observations:
- Creating the service with scale=3 is fast, scaling to 4/5 (creating containers on the new nodes) takes half a minute or so
- Exposing container ports on the new hosts directly works properly
- Just now, after removing, re-adding, and then scaling the service, it works better, one of the new nodes replies, the other one only sometimes.
- iptables look identical on all nodes.
What could be the cause of this? How can I debug it?
