0

I am trying to configure Outlier Detection for a consul connect service mesh based on this documentation.

https://learn.hashicorp.com/tutorials/consul/service-mesh-circuit-breaking?in=consul/developer-mesh

The documentation shows that Outlier Detection and Circuit breaking can be configured using the config stanza inside proxy.upstreams. But the following job file throws error - Blocks of type "config" are not expected here.

job "docs" {
  datacenters = ["dc1"]

  group "docs" {
    network {
      mode = "bridge"
    }
    service {
      name = "docs"
      port = "5678"

      connect {
        sidecar_service {
          proxy {
            upstreams {
              destination_name = "demo"
              local_bind_port  = 10082
              config {
                connect_timeout_ms = 3000
              }
            }
          }
        }
      }
    }
    task "server" {
      driver = "docker"

      config {
        image = "hashicorp/http-echo"
        args = [
          "-listen",
          ":5678",
          "-text",
          "hello world",
        ]
      }
    }
  }
}

Am I doing anything wrong? Is this not the right way to configure circuit breaking in nomad job file?

Jawahar
  • 4,775
  • 1
  • 24
  • 47

1 Answers1

0

sidecar Proxy, Circuit breaking, ingress, egress must be implemented with consul directly and not from nomad. Also, In your job you didn't map the port inside docker and outside port. consul work a specific version of envoy load balacner.

  1. First launch your job without connect stanza and do port mapping

  2. install envoy and do proxy connect connection manually to test

  3. once test work make a service proxy to launch your sidecar your circuit breaking

1- Launching job: (by exemple your port inside docker is 8080 )

job "docs" {
  datacenters = ["dc1"]

  group "docs" {
    network {
      mode = "bridge"
    }
    
    task "server" {
      driver = "docker"

      config {
        image = "hashicorp/http-echo"
        args = [
          "-listen",
          ":5678",
          "-text",
          "hello world",
        ]
        port_map {
            docs = 8080
        }
      }
      resources {
           network {
               mbits = 10
               port "docs" { static = 5678 }
           }
      }
  service {
    name = "docs"
    port = "docs"
    check {
      name     = "docs port alive"
      type     = "http"
      path     = "/"
      interval = "10s"
      timeout  = "2s"
    }
  }
    }
  }
}

2-check your consul version and install supported envoy version here. i use consul 1.11 so i will install supported envoy 1.18.4

yum -y -q install tar
curl https://func-e.io/install.sh | bash -s -- -b /usr/local/bin
func-e use 1.18.4

make the envoy bin available

cp /root/.func-e/versions/1.18.4/bin/envoy /usr/local/bin/

Proxy integration

insert at your end of consul config .for me my config are stored in

/etc/consul.d/config.hcl

config_entries {
  bootstrap = [
    {
      kind = "proxy-defaults"
      name = "global"
      config {
        protocol                   = "http"
      }
    }
  ]
}

**restart your consul service to check if envoy proxy integration worked**

systemctl restart consul

Overwrite your service registration in consul with consul file :

cat > /etc/consul.d/docs.hcl <<- EOF

service {
  name = "docs"
  port = 5678
  #token = "" # put api service token here
  check {
    id = "docs"
    name = "HTTP API on Port 5678"
    http = "http://localhost:5678"
    interval = "30s"
  }
  connect {
    sidecar_service {
      port = 20000
      check {
        name     = "Connect Envoy Sidecar"
        tcp      = "127.0.0.1:20000"
        interval = "10s"
      }
    }
  }
}
EOF

restart service consul or reload it

systemctl restart consul

Test proxy side car working

consul connect envoy -sidecar-for=docs

create docs service proxy Create at /etc/systemd/system/consul-envoy-docs.service and input the following:

cat > /etc/systemd/system/consul-envoy.service <<- EOF
[Unit]
Description=Consul Envoy
After=syslog.target network.target 
[Service]
ExecStart=/usr/local/bin/consul connect envoy -sidecar-for=docs
ExecStop=/bin/sleep 5
Restart=always

[Install]
WantedBy=multi-user.target
EOF

Restart consul and start consul-envoy:

systemctl daemon-reload
systemctl restart consul
systemctl start consul-envoy-docs

In the event that consul-envoy fails, restart it with:

systemctl restart consul-envoy

3. Well if all work correctly , adapt conf file in /etc/systemd/system/consul-envoy-docs.service as described here to make circuit breaking

If someone have issue with nomad , consul , vault , envoy or hashistack tag me
GhislainAdon
  • 31
  • 7