I have setup a 3-node Docker Swarm. I can ping but cannot curl a docker service that ends up deployed on Host B from within a docker container on Host A. What am I missing? What else can I do to troubleshoot?
The host environment: one manager, two workers. Provisioned via OpenStack (it's got an element of virtualization in the infrastructure behind the scenes). The swarm is running on CentOS 7. SELinux is disabled. the docker services are all on an overlay network via:
docker network create --driver=overlay --opt com.docker.network.driver.mtu=1200 --attachable proxy-overlay
the manager node via:
docker swarm init --advertise-addr <manager-public-ip> --data-path-port 8091
and joining via the workers via:
docker swarm join --token <token-from-init> --advertise-addr <worker-ip> <manager-public-ip>:2377
Here are the different things I've attempted and i'd expect all of them to work. point 6 and 7 are what I'm trying to resolve.
- I can ping Host A to Host B
- I can do a curl from Host A to Host B
- I can ping from inside a container running on Host A to Host B
- I can curl from inside a container running on Host A to Host B
- I can ping a docker service from within a container running on Host A that DNS resolves to a container running on Host B
- I cannot curl a docker service from within a container running on Host A that DNS resolves to a container running on Host B
- I cannot curl the DNS resolved IP of the container running on Host B (point 5) from within a container running on Host A
- I can curl the docker service from another docker container IFF containers are on the same host
The end error ends up being:
curl: (7) Failed to connect to port 80: Connection timed out
Some areas I've explored:
- Dropping all iptable rules
- Lowering the MTU to 1200 (set in the /etc/docker/daemon.json and then manually creating
docker_gwbridgenetwork via steps outlined in docker's documentation.) - Ensuring Security group has ingress/egress open (OpenStack)
- Ensured MTU changes are propagated on all nodes (so it's not getting dropped at the network level
- Ensure overlay network is correct/propogated on all nodes
- Matched IPs from steps above to that from various docker inspects.
- I saw some virtual infrastructures sometimes use port 4789 for communications, so I changed the docker port to 8091, but have tried both ports.
If time permits, I'll go wrangle up and link the several dozen github issue pages and docker documentation web pages I've referenced in above information.
