1

I'm trying to get Consul Connect side car envoy to work but the health checks for sidecar keeps failing.

I'm using following versions of Consul and Nomad

Consul : 1.7.3
Nomad : 0.11.1
CNI Plugins : 0.8.6

My setup looks like follows.

1 Consul Server running consul in docker container.

docker run -d --net=host --name=server -v /var/consul/:/consul/config consul:1.7 agent -server -ui -node=server-1 -bind=$internal_ip -ui -bootstrap-expect=1 -client=0.0.0.0

internal_ip is the internal IP address of my GCP VM.

1 Nomad Server with Consul Agent in client mode

nohup nomad agent -config=/etc/nomad.d/server.hcl &

docker run -d --name=consul-client --net=host -v ${volume_path}:/consul/config/ consul:1.7 agent -node=$node_name -bind=$internal_ip -join=${server_ip} -client=0.0.0.0

interal_ip is the internal IP address of GCP VM and server_ip is the internal IP address of Server VM.

2 Nomad Client with Consul Agent in client mode

nohup nomad agent -config=/etc/nomad.d/client.hcl &

 docker run -d --name=consul-client --net=host -v ${volume_path}:/consul/config/ consul:1.7 agent -node=$node_name -bind=$internal_ip -join=${server_ip} -client=0.0.0.0

On Nomad clients, I also have consul binary available in path.

Now I'm trying to deploy the sample Nomad and Consul Connect job from here

    job "countdash" {
      datacenters = ["dc1"]

      group "api" {
        network {
          mode = "bridge"
        }

        service {
          name = "count-api"
          port = "9001"

          connect {
            sidecar_service {}
          }
        }

        task "web" {
          driver = "docker"

          config {
            image = "hashicorpnomad/counter-api:v1"
          }
        }
      }

      group "dashboard" {
        network {
          mode = "bridge"

          port "http" {
            static = 9002
            to     = 9002
          }
        }

        service {
          name = "count-dashboard"
          port = "9002"

          connect {
            sidecar_service {
              proxy {
                upstreams {
                  destination_name = "count-api"
                  local_bind_port  = 8080
                }
              }
            }
          }
        }

        task "dashboard" {
          driver = "docker"

          env {
            COUNTING_SERVICE_URL = "http://${NOMAD_UPSTREAM_ADDR_count_api}"
          }

          config {
            image = "hashicorpnomad/counter-dashboard:v1"
          }
        }
      }
    }

The docker container for service and sidecar gets started and gets registered in Consul, but I'm unable to access any of the service.

enter image description here

I SSH onto the Nomad Client node and can see the container running.

enter image description here

  1. Odd thing I noticed is that I cannot see port forwarded to the host
  2. I cannot access it via curl from host.

I tried doing curl $internal_ip:9002 but it didn't work.

I checked if Nomad created any new bridge network since that's what I used as mode in the network stanza but there are no new networks.

enter image description here

Is there anything that I'm missing in my setup ?

kaysush
  • 4,797
  • 3
  • 27
  • 47
  • have you checked the nomad logs from your docker container? I think consul connect needs to edit iptables rules and your container doesn't have `--cap-add=NET_ADMIN`, that should be failing and maybe there are other errors in the nomad logs? – maxm May 28 '20 at 17:24
  • @maxm I'll try that. – kaysush May 29 '20 at 05:09

1 Answers1

1

Have you tried setting COUNTING_SERVICE_URL = "http://${NOMAD_UPSTREAM_ADDR_count_api}" to COUNTING_SERVICE_URL = "http://localhost:8080", since that is the local bind port that the envoy proxy will be listening on to forward traffic to the count-api.

An example of a working connect setup can be found at https://github.com/hashicorp/video-content/tree/master/nomad-connect-integration/nomad_jobs

Erik Veld
  • 11
  • 4