Rabbitmq cluster on Nomad

Question

I want to create a rabbitmq cluster in a nomad cluster. Right now I can run rabbitmq docker containers, but I can't create a cluster based on consul discovery. After containters starts i got this massages on every node:

[warning] <0.229.0> Could not auto-cluster with node rabbit@10.14.1.112: {badrpc,nodedown}
[warning] <0.229.0> Could not auto-cluster with node rabbit@test-nomad1: {badrpc,nodedown}
[warning] <0.229.0> Could not auto-cluster with node rabbit@test-nomad2: {badrpc,nodedown}
[warning] <0.229.0> Could not auto-cluster with node rabbit@test-nomad3: {badrpc,nodedown}

Where is my problem? I'm thinking maybe one node needs to be started first and I've added one container with prestart , but that won't help me. My nomad job:

job "test" {
datacenters = ["dc1"]

group "test-group" {
count = 4
update {
  max_parallel = 1
}

migrate {
  max_parallel = 1
  health_check = "checks"
  min_healthy_time = "30s"
  healthy_deadline = "60s"
}
task "rabbit" {
  driver = "docker"
  config {
    image = "<my_registry>/rabbitmq:consul"
    force_pull = true
    hostname = "${attr.unique.hostname}"
    port_map {
      amqp = 5672
      ui = 15672
      discovery = 4369
      clustering = 25672
    }
    auth {
      server_address = "<my_registry>"
      username = "user"
      password = "pass"
    }
  }

  env {
    RABBITMQ_ERLANG_COOKIE = "RabbitMQ"
    RABBITMQ_DEFAULT_USER = "test"
    RABBITMQ_DEFAULT_PASS = "test"
    CONSUL_HOST = "10.14.1.109"
    CONSUL_SVC_PORT = "${NOMAD_HOST_PORT_amqp}"
    CONSUL_SVC_TAGS = "amqp"
  }

  resources {
    network {
      port "amqp" { static = 5672 }
      port "ui" { static = 15672 }
      port "discovery" { static = 4369 }
      port "clustering" { static = 25672 }
    }
  }

  service {
    name = "rabbitmq"
    port = "ui"
    tags = ["management", "http"]
  }
}
}
}

Rabbitmq config:

loopback_users.guest = false
listeners.tcp.default = 5672
hipe_compile = false
management.listener.port = 15672
management.listener.ssl = false
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_consul
cluster_formation.consul.acl_token = <my_token>
cluster_formation.consul.host = 10.14.1.109
cluster_formation.consul.svc_addr_auto = true
cluster_formation.consul.svc_addr_use_nodename = false

In consul services i see rabbit Thanks for any help!

score 0 · Accepted Answer · answered Feb 02 '23 at 06:30

0

Problem fixed, docker container not configured to get consul dns. How to fix it https://awesomeprogrammer.com/blog/2021/12/22/consul-dns-services-discovery-in-docker-gotchas/

answered Feb 02 '23 at 06:30

sflyer

1
3

Great that you were able to solve it. But you should include the gist of that blog post into your answer, otherwise the answer becomes useless when that URL dies. – Gerald Schneider Feb 02 '23 at 06:34

Rabbitmq cluster on Nomad

1 Answers1