How to set up Cassandra Docker cluster in Marathon with BRIDGE network?

Question

I have a production DC/OS(v1.8.4) cluster and I am trying to setup a Cassandra cluster inside it. I use Marathon(v1.3.0) to deploy Cassandra nodes. I use the official Docker image of Cassandra and more specifically the 2.2.3 version.

First Case: Deploy Cassandra using HOST mode network - Everything OK

In this case, I first deploy a node that I call cassasndra-seed and it attaches to a physical host with IP 10.32.0.6. From the stdout log of Marathon for this service I can see that "Node /10.32.0.6 state jump to normal" and that listen_address and broadcast_address are set to 10.32.0.6. If I check the mesos-dns records using "_cassandra-seed._tcp.marathon.mesos SRV" in a master node I can see that the IP that resolves for this service is 10.32.0.6. The node is fully functional and I manage to create a test database.

{
  "id": "/cassandra-seed",
  "cpus": 1.5,
  "mem": 8192,
  "disk": 0,
  "instances": 1,
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "cassandra:2.2.3",
      "network": "HOST",
      "ports": [7199,7000,7001,9160,9042],
      "requirePorts": true,
      "privileged": true
    }
  },
  "constraints": [ ["hostname","UNIQUE"] ],
  "env": { "CASSANDRA_CLUSTER_NAME": "democluster" }
}

Now I add one more node of cassandra using a separate deployment and providing 10.32.0.6 as seed (set "CASSANDRA_SEEDS": "10.32.0.6" in the env section of the deployment JSON). The new node gets the IP of another physical host (same pattern as before) and manages to gossip with the seed node. Thus, we have a functioning Cassandra cluster.

{
  "id": "/cassandra",
  "cpus": 1.5,
  "mem": 8192,
  "disk": 0,
  "instances": 1,
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "cassandra:2.2.3",
      "network": "HOST",
      "ports": [7199,7000,7001,9160,9042],
      "requirePorts": true,
      "privileged": true
    }
  },
  "constraints": [ ["hostname","UNIQUE"] ],
  "env": { 
    "CASSANDRA_CLUSTER_NAME": "democluster",
    "CASSANDRA_SEEDS": "10.32.0.6" 
  }
}

Second Case: Deploy Cassandra using BRIDGE mode network - Houston we have a problem

In this case, I also deploy a first cassandra-seed node and it attaches to a physical host with IP 10.32.0.6. However, now at the stdout log of the service in Marathon I can see that "Node /172.17.0.2 state jump to normal" and that listen_address and broadcast_address are set to 172.17.0.2. 172.17.0.2 is the IP of the docker container (found using docker inspect). However, if I check the mesos-dns records using "_cassandra-seed._tcp.marathon.mesos SRV" in a master node I can see that the IP that resolves for this service is 10.32.0.6. The node is fully functional and I manage to create a test database.

{
  "id": "/cassandra-seed",
  "cpus": 1.5,
  "mem": 8192,
  "disk": 0,
  "instances": 1,
  "container": {
    "type": "DOCKER",
      "docker": {
      "image": "cassandra:2.2.3",
      "network": "BRIDGE",
      "portMappings": [
        {"containerPort": 7000, "hostPort": 7000, "servicePort": 0 },
        {"containerPort": 7001, "hostPort": 7001, "servicePort": 0 },
        {"containerPort": 7199, "hostPort": 7199, "servicePort": 0 },
        {"containerPort": 9042, "hostPort": 9042, "servicePort": 0 },
        {"containerPort": 9160, "hostPort": 9160, "servicePort": 0 },
      ],
      "privileged": true,
    }
  },
  "constraints": [ [ "hostname", "UNIQUE" ] ],
  "env": {"CASSANDRA_CLUSTER_NAME": "democluster"}
}

Now I add one more node of cassandra using a separate deployment and providing 10.32.0.6 as seed. The new node attaches to another host and gets the IP of his container (Node /172.17.0.2 state jump to normal). The result is that the new node cannot gossip with the seed.

{
  "id": "/cassandra",
  "cpus": 1.5,
  "mem": 8192,
  "disk": 0,
  "instances": 1,
  "container": {
    "type": "DOCKER",
      "docker": {
      "image": "cassandra:2.2.3",
      "network": "BRIDGE",
      "portMappings": [
        {"containerPort": 7000, "hostPort": 7000, "servicePort": 0 },
        {"containerPort": 7001, "hostPort": 7001, "servicePort": 0 },
        {"containerPort": 7199, "hostPort": 7199, "servicePort": 0 },
        {"containerPort": 9042, "hostPort": 9042, "servicePort": 0 },
        {"containerPort": 9160, "hostPort": 9160, "servicePort": 0 },
      ],
      "privileged": true,
    }
  },
  "constraints": [ [ "hostname", "UNIQUE" ] ],
  "env": { 
    "CASSANDRA_CLUSTER_NAME": "democluster",
    "CASSANDRA_SEEDS": "10.32.0.6" 
  }
}

The question is how could I make the two nodes gossip in the second case? Which is the IP that I should provide as seed to the second node in order to find the first one? The 172.17.0.2 is the docker container IP and cannot be reached by the second node. For example, could cassandra instance in the seed node get the IP of the physical host just like in the host network mode?

Thank you in advance!

Answer 1

When forming a cassandra cluster in bridge network mode below settinhs should be taken care. 1. Set below values to host Ip (not container ip)

Seeds : public_ip Broadcast_address : public_ip Broadcast_rpc_address : public_ip

2. Set listen_address to container Ip

Listen_address : 172.17.x.x

3 . Set rpc_address to 0.0.0.0 (don't use localhost)

This way we can actually form a Cassandra cluster using bridge network.

Give it a try. Make sure required ports should be accessible from outside world.