Docker network: Fail to adress containers by name on a multihost network

Question

I have 3 hosts running has a swarm cluster, with an overlay network over it using docker 1.11

Machines are started using boot2docker and virtualbox:

docker-machine create -d virtualbox af-consul
docker run -d -p "8500:8500" -h "consul" progrium/consul -server -bootstrap

docker-machine create -d virtualbox --virtualbox-memory 4096 \
--swarm \
--swarm-discovery="consul://$(docker-machine ip af-consul):8500" \
--engine-opt="cluster-store=consul://$(docker-machine ip af-consul):8500" \
--engine-opt="cluster-advertise=eth0:2376" af-repo

docker-machine create -d virtualbox --virtualbox-memory 4096 --swarm --swarm-master \
--swarm-discovery="consul://$(docker-machine ip af-consul):8500" \
--engine-opt="cluster-store=consul://$(docker-machine ip af-consul):8500" \
--engine-opt="cluster-advertise=eth0:2376" af-jenkins

Network is created like that:

eval $(docker-machine env --swarm af-jenkins)
docker network create -d overlay appfactory_overlay

Then deployment of containers is done using compose:

version: '2'
jenkins:
    user: root
    restart: always
    image: af-repo:5000/jenkins:latest
    # build: ./jenkins
    container_name: jenkins
    ports:
        - "8080:8080"
        - "50000:50000"
    networks:
        - appfactory_overlay
    volumes_from:
        - container:jenkins_data:rw
    environment:
      - "constraint:node==af-jenkins"
    external_links:
        - jenkins_data

    docker_registry:
        user: root
        container_name: docker_registry
        restart: always
        build: ./
        ports:
          - 5000:5000
        networks:
            - appfactory_overlay
        environment:
          REGISTRY_HTTP_TLS_CERTIFICATE: /etc/ssl/certs/domain.crt
          REGISTRY_HTTP_TLS_KEY: /etc/ssl/certs/domain.key
          constraint: "node==af-repo"

    reports:
        user: root
        restart: always
        image: httpd:2.4
        container_name: reports
        ports:
        - "2222:22"
        - "80:80"
        volumes_from:
            - container:repository_data:rw
        networks:
            - appfactory_overlay
        environment:
          - "constraint:node==af-repo"

networks:
  appfactory_overlay:
    external:
      name: appfactory_overlay

When I ssh to af-repo and exec a /bin/bash on reports container I can ping docker_registry container, which is on the same host:

root@74068f00ffdb:/usr/local/apache2# ping docker_registry
PING docker_registry (10.0.0.2): 56 data bytes
64 bytes from 10.0.0.2: icmp_seq=6 ttl=64 time=0.157 ms
64 bytes from 10.0.0.2: icmp_seq=7 ttl=64 time=0.126 ms
64 bytes from 10.0.0.2: icmp_seq=8 ttl=64 time=0.127 ms
64 bytes from 10.0.0.2: icmp_seq=9 ttl=64 time=0.132 ms

But when I try to ping jenkins container, which is on another host:

root@74068f00ffdb:/usr/local/apache2# ping jenkins
PING jenkins (10.0.0.5): 56 data bytes
92 bytes from 74068f00ffdb (10.0.0.4): Destination Host Unreachable
92 bytes from 74068f00ffdb (10.0.0.4): Destination Host Unreachable
92 bytes from 74068f00ffdb (10.0.0.4): Destination Host Unreachable
92 bytes from 74068f00ffdb (10.0.0.4): Destination Host Unreachable
92 bytes from 74068f00ffdb (10.0.0.4): Destination Host Unreachable
92 bytes from 74068f00ffdb (10.0.0.4): Destination Host Unreachable
92 bytes from 74068f00ffdb (10.0.0.4): Destination Host Unreachable
92 bytes from 74068f00ffdb (10.0.0.4): Destination Host Unreachable
92 bytes from 74068f00ffdb (10.0.0.4): Destination Host Unreachable
92 bytes from 74068f00ffdb (10.0.0.4): Destination Host Unreachable
92 bytes from 74068f00ffdb (10.0.0.4): Destination Host Unreachable
92 bytes from 74068f00ffdb (10.0.0.4): Destination Host Unreachable

Network seems to be correct when I inspect it:

docker@af-repo:~$ docker network inspect appfactory_overlay
[
    {
        "Name": "appfactory_overlay",
        "Id": "2cedd8b85bd302649123c3313e6e2beb5ddf3dc8265851fda1318f4b048dd795",
        "Scope": "global",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "10.0.0.0/24",
                    "Gateway": "10.0.0.1/24"
                }
            ]
        },
        "Internal": false,
        "Containers": {
            "74068f00ffdb19135b605eb84ff6a23887a44dd03a3f9645a130fb03ad074312": {
                "Name": "reports",
                "EndpointID": "78503be0001b75b60e16f788009c1d5bb573011582ca828fbc3acd8ab3c67459",
                "MacAddress": "02:42:0a:00:00:04",
                "IPv4Address": "10.0.0.4/24",
                "IPv6Address": ""
            },
            "ep-eb61b033ae6e073c2634d36a8511434bbe26615e9c18df1c7545515b4f3a8c05": {
                "Name": "web",
                "EndpointID": "eb61b033ae6e073c2634d36a8511434bbe26615e9c18df1c7545515b4f3a8c05",
                "MacAddress": "02:42:0a:00:00:03",
                "IPv4Address": "10.0.0.3/24",
                "IPv6Address": ""
            },
            "ep-ee9feed770ef9f7684981e848a41de6bd67632f716672f8ed279dddc15afcc08": {
                "Name": "jenkins",
                "EndpointID": "ee9feed770ef9f7684981e848a41de6bd67632f716672f8ed279dddc15afcc08",
                "MacAddress": "02:42:0a:00:00:05",
                "IPv4Address": "10.0.0.5/24",
                "IPv6Address": ""
            },
            "fc125785613bf07202d2a0f39371394890bd04f7e5c4e34353ecc55e1e5d6b0b": {
                "Name": "docker_registry",
                "EndpointID": "6cf8d868c299d5b442abd5f370e1392a34585dede221cff19b95edb29baefaaa",
                "MacAddress": "02:42:0a:00:00:02",
                "IPv4Address": "10.0.0.2/24",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]

Output is the same when doing it from af-jenkins. So from what I understand, my network isn't working at all as my ability to ping a container from another one on the same host come from the automatically created bridge network for those containers.

Any help greatly appreciated :)

Edit: Here is the cluster info output

        { AppFactory } master » docker info                                                                                                                                                                                   /d/Alstom/AppFactory
    Containers: 12
     Running: 8
     Paused: 0
     Stopped: 4
    Images: 27
    Server Version: swarm/1.2.4
    Role: primary
    Strategy: spread
    Filters: health, port, containerslots, dependency, affinity, constraint
    Nodes: 3
     af-jenkins: 192.168.99.102:2376
      └ ID: BAG4:XIOG:JXLS:7UAO:JHFT:NOAY:OW35:L2RK:QV37:OQ2H:G5QN:KR4D
      └ Status: Healthy
      └ Containers: 4 (3 Running, 0 Paused, 1 Stopped)
      └ Reserved CPUs: 0 / 1
      └ Reserved Memory: 0 B / 4.051 GiB
      └ Labels: kernelversion=4.4.16-boot2docker, operatingsystem=Boot2Docker 1.12.0 (TCL 7.2); HEAD : e030bab - Fri Jul 29 00:29:14 UTC 2016, provider=virtualbox, storagedriver=aufs
      └ UpdatedAt: 2016-07-30T10:15:40Z
      └ ServerVersion: 1.12.0
     af-repo: 192.168.99.101:2376
      └ ID: 5U2Z:46IU:CRAG:V3S7:Q5WF:TESM:KBRN:H7L3:7LZD:H4DX:DTEC:WXNE
      └ Status: Healthy
      └ Containers: 6 (3 Running, 0 Paused, 3 Stopped)
      └ Reserved CPUs: 0 / 1
      └ Reserved Memory: 0 B / 4.051 GiB
      └ Labels: kernelversion=4.4.16-boot2docker, operatingsystem=Boot2Docker 1.12.0 (TCL 7.2); HEAD : e030bab - Fri Jul 29 00:29:14 UTC 2016, provider=virtualbox, storagedriver=aufs
      └ UpdatedAt: 2016-07-30T10:15:45Z
      └ ServerVersion: 1.12.0
     af-web: 192.168.99.103:2376
      └ ID: HJ7O:62JN:6RVT:4UKL:AAJ3:6VWU:2LGF:EIYC:CJAH:F4RJ:GNCF:J3TR
      └ Status: Healthy
      └ Containers: 2 (2 Running, 0 Paused, 0 Stopped)
      └ Reserved CPUs: 0 / 1
      └ Reserved Memory: 0 B / 517.3 MiB
      └ Labels: kernelversion=4.4.16-boot2docker, operatingsystem=Boot2Docker 1.12.0 (TCL 7.2); HEAD : e030bab - Fri Jul 29 00:29:14 UTC 2016, provider=virtualbox, storagedriver=aufs
      └ UpdatedAt: 2016-07-30T10:15:18Z
      └ ServerVersion: 1.12.0
    Plugins:
     Volume:
     Network:
    Swarm:
     NodeID:
     Is Manager: false
     Node Address:
    Security Options:
    Kernel Version: 4.4.16-boot2docker
    Operating System: linux
    Architecture: amd64
    CPUs: 3
    Total Memory: 8.608 GiB
    Name: 8759d85be147
    Docker Root Dir:
    Debug Mode (client): false
    Debug Mode (server): false
    WARNING: No kernel memory limit support

When using `docker-machine,` Virtualbox's "host only" interface is `eth1`, try changing your `cluster-advertise` settings to use `eth1` instead. — slugonamission, Jul 30 '16 at 10:17

score 0 · Answer 1 · answered Jul 29 '16 at 18:50

0

I think, your problem is that you are not pointing your docker CLI to the cluster. Before to create the overlay network you have to connect to the swarm cluster using the flag --swarm.

eval $(docker-machine env --swarm af-jenkins)

If this is not the problem, let us know what is the output of docker info command in your cluster.

Note: With docker 1.12.0 you can simplify the process of create a cluster.

answered Jul 29 '16 at 18:50

Hemerson Varela

24,034
16
68
69

Sadly this is not the issue, I've updated my question as the network is created exactly like you suggest. Also added the docker info output of af-jenkins swarm master. I know about docker 1.12 which I'll use soon. – Dealus Jul 30 '16 at 10:14

score 0 · Accepted Answer · answered Jul 30 '16 at 11:08

0

When using docker-machine, Virtualbox's "host only" interface is eth1, try changing your cluster-advertise settings to use eth1 instead. – slugonamission

Thanks.

answered Jul 30 '16 at 11:08

Dealus

25
4

Docker network: Fail to adress containers by name on a multihost network

2 Answers2