1

Should I run consul slaves alongside nomad slaves or inside them? The later might not make sense at all but I'm asking it just in case.

I brought my own nomad cluster up with consul slaves running alongside nomad slaves (inside worker nodes), my deployable artifacts are docker containers (java spring applications). The issue with my current setup is that my applications can't access consul slaves (to read configurations) (none of 0.0.0.0, localhost, worker node ip worked)

Lets say my service exposes 8080, I configured docker part (in hcl file) to use bridge as network mode. Nomad maps 8080 to 43210. Everything is fine until my service tries to reach the consul slave to read configuration. Ideally giving nomad worker node IP as consul host to Spring should suffice. But for some reason it's not.

I'm using latest version of nomad.

I configured my nomad slaves like https://github.com/bmd007/statefull-geofencing-faas/blob/master/infrastructure/nomad/client1.hcl

And the link below shows how I configured/ran my consul slave: https://github.com/bmd007/statefull-geofencing-faas/blob/master/infrastructure/server2.yml

Note: if I use static port mapping and host as the network mode for docker (in nomad) I'll be fine but then I can't deploy more than one instance of each application in each worker node (due to port conflic)

Mahdi Amini
  • 402
  • 1
  • 3
  • 17
  • 1
    Have you tried using `172.17.0.1` from within docker container deployed with nomad? – Ilya Kisil Jan 05 '21 at 14:35
  • 2
    what is a "slave" here? consul has clients and servers, do you mean clients? https://www.consul.io/docs/architecture. Not trying to be pedantic, just not sure what's going on. – maxm Jan 05 '21 at 18:16
  • 1
    Ah, I understand. the server2.yml is showing how you are running your consul "server" right? and your nomad job for your consul clients is not shown? – maxm Jan 05 '21 at 18:18
  • 1
    I would recommend running the consul nomad job as a "system" job and using the static port map, as you suggest. We do this for a handful of things, and while it's true that you'll never be able to host another job with port 8500, I think it's more elegant that your services can always reach consul at machine_ip:8500 – maxm Jan 05 '21 at 18:21

3 Answers3

1

Nomad jobs listen on a specific host/port pair.

enter image description here

You might want to ssh into the server and run docker ps to see what host/port pair the job is listening on.

a93c5cb46a3e   image-name bash   2 hours ago    Up 2 hours  10.0.47.2:21435->8000/tcp, 10.0.47.2:21435->8000/udp foo-bar

Additionally, you will need to ensure that the consul nomad job is listening on port 0.0.0.0, or the specific ip of the machine. I believe that is this config value: https://www.consul.io/docs/agent/options.html#_bind

All those will need to match up in order to consul to be reachable.

More generally, I might recommend: if you're going to run consul with nomad, you might want to switch to host networking, so that you don't have to deal with the specifics of the networking within a container. Additionally, you could schedule consul as a system job so that it is automatically present on every host.

maxm
  • 3,412
  • 1
  • 19
  • 27
  • I will try it and will come back with the results. Consul as nomad system job was something missing in my understanding of system. – Mahdi Amini Jan 05 '21 at 20:20
1

So I managed to solve the issue like this:

  1. nomad.job.group.network.mode = host
  2. nomad.job.group.network.port: port "http" {}
  3. nomad.job.group.task.driver = docker
  4. nomad.job.group.task.config.network_mode = host
  5. nomad.job.group.task.config.ports = ["http"]
  6. nomad.job.group.task.service.connect: connect { native = true }
  7. nomad.job.group.task.env: SERVER_PORT= "${NOMAD_PORT_http}"
  8. nomad.job.group.task.env: SPRING_CLOUD_CONSUL_HOST = "localhost"
  9. nomad.job.group.task.env: SPRING_CLOUD_SERVICE_REGISTRY_AUTO_REGISTRATION_ENABLED = "false"
  10. Running consul agent (slaves) using docker-compose alongside nomad agent (slave) with host as network mode + exposing all required ports.

Example of nomad job: https://github.com/bmd007/statefull-geofencing-faas/blob/master/infrastructure/nomad/location-update-publisher.hcl

Example of consul agent config (docker-compose file): https://github.com/bmd007/statefull-geofencing-faas/blob/master/infrastructure/server2.yml

Mahdi Amini
  • 402
  • 1
  • 3
  • 17
1

Disclaimer: The LAB is part of Cluster Visualization Framework called: LiteArch Trafik which I have created as an interesting exercise to understand Nomad and Consul.

It took me long time to shift my mind from K8S to Nomad and Consul, Integration them was one of my effort I spent in the last year. When service resolution doesn't work, I found out it's more or less the DNS configuration on servers.

There is a section for it on Hashicorp documentation called DNS Forwarding Hashicorp DNS Forwarding

I have created a LAB which explains how to set up Nomad and Consul. But you can use the LAB seperately. I created the LAB after learning the hard way how to install the cluster and how to integrate Nomad and Consul. With the LAB you need Ubuntu Multipass installed. You execute one script and you will get full functional Cluster locally with three servers and three nodes. It shows you as well how to install docker and integrate the services with Consul and DNS services on Ubuntu. After running the LAB you will get the links to Nomad, Fabio, Consul. Hopefully it will guide you through the learning process of Nomad and Consul

LAB: LAB Trafik:Trafik Visualizer

Maher
  • 41
  • 3