Cannot ping container in another host with docker consul

Question

I have 3 CentoS 7 machines. I tried to set up a cassandra cluster. A cassandra node can discover all other nodes if they are in the same network, therefore I was looking for a way to set up 2 containers on different hosts but still make them work as if they are in the same network.

I tried doing it with consul in the following steps:

1. Creating a consul container on machine0:

   docker run -d -p 8500:8500 -h consul --name consul progrium/consul -server -bootstrap

2. Adding the consul configuration to /lib/systemd/system/docker.service in machine1 and machine2:

   -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock --cluster-store=consul://*machine0-external-ip*:8500/network --cluster-advertise=machine0-external-ip:2375 --userland-proxy=false \

3. Creating an overlay network docker network create -d overlay --subnet=10.10.10.0/24 testNetwork

4. Creating busybox containers on both machine1 and machine2, with my test network machine1- docker run -itd --name container1 --net testNetwork busybox machine2- docker run -itd --name container2 --net testNetwork busybox

5. From either machine (1 or 2 ) I could see the containers in the network through this command docker network inspect testNetwork resulting in :

   {
   "Name": "testNetwork",
   "Id": "12020d9c66762dcf4db74cb44ffed2fe8f9e3eb531fe5e4e3f16640185154447",
   "Scope": "global",
   "Driver": "overlay",
   "EnableIPv6": false,
   "IPAM": {
       "Driver": "default",
       "Options": {},
       "Config": [
           {
               "Subnet": "10.10.10.0/24"
           }
       ]
   },
   "Internal": false,
   "Containers": {
       "c704cdedf845507891ea25ece4536e35a8984ffc98850d44fd6520e1954a203a": {
           "Name": "container1",
           "EndpointID": "345a3895f6480eecaa3e03f83ff021197a3241ebedca5cb635c06f8a83d259d6",
           "MacAddress": "02:42:0a:0a:0a:02",
           "IPv4Address": "10.10.10.2/24",
           "IPv6Address": ""
       },
       "ep-c507d3c94db20b519da28319bcb07e63297d2f2c12c0fdd52e88807bbb255743": {
           "Name": "container2",
           "EndpointID": "c507d3c94db20b519da28319bcb07e63297d2f2c12c0fdd52e88807bbb255743",
           "MacAddress": "02:42:0a:0a:0a:03",
           "IPv4Address": "10.10.10.3/24",
           "IPv6Address": ""
       }
   },
   "Options": {},
   "Labels": {}
   

   }

6. Trying to ping container2 from inside container1 docker exec container1 ping -w 3 container2. This results in

   PING container2 (10.10.10.3): 56 data bytes
   
   --- container2 ping statistics --- 3 packets transmitted, 0 packets 
   received, 100% packet loss
   

You can see the ping is going to the subnet address (10.10.10.3) which is what I want.

What am I doing wrong?

To my understanding the container is supposed to go through the autocreated network docker_gwbridge. In order to know where does this subnet is for real (which is on a different machine) I ran a sniffer on the packets and it seems like the container just looks up that address in his inner network and doesn't even go through the docker_gwbridge...

If anyone has a different approach or suggestions to my problem i would love to hear it!

Thanks in advance!

Answer 1

I fail to understand what you are trying to acheive here with consul, because you clearly are not using it in your approach. You can look at my answer on the following post to get more clarity on it:

Does my docker images need its own consul client instances?

Hope it gives some clarity.