How to deploy distributed h2o flow cluster with docker?

Question

I'm able to deploy a h2o cluster with ec2 instances and having the private ip in the flatfile. Doing the same with docker works but I can't figure out what to enter into the flatfile so they can create the cluster. Private IP the container is running on is not working

Answer 1

Can the containers ping each others ip?

When launching h2o are you forcing the interface to use the container ip? java -jar h2o.jar -flatfile flatfile -ip -port

Are these docker containers when run exposing the port 54321 to each other? docker run -it -p 54321:54321

Answer 2

As of the current version of H2O (3.18.0.8) what you can do is:

1) Put the list of EC2 instance private IP and port in the flatfile:

private-ip-1:54321
private-ip-2:54321
private-ip-3:54321

private-ip-1 should be in the standard four octet a.b.c.d network address form.

2) provide the --network host option to docker run:

docker run --network host [... rest of run command ...]

If you don't do this, what I think is happening is that each local H2O instance gets confused trying to figure out which node in the flatfile is itself. Since none of the local interfaces match what's in the flatfile, the cluster formation doesn't work for whatever reason.

I would actually consider this a bug.

Answer 3

Ultimately, the solution for running H2O in docker may be to use a network plugin like weave, because weave can use multicasting (unlike docker overlay).

But I managed to hack together a solution for running H2O in docker swarm on an overlay network and a flatfile. The issue with running in swarm is that docker assigns each H2O instance two IP addresses: one resolvable as the stack_service and the other seen as $HOSTNAME from within the instance. H2O needs to use the $HOSTNAME IP, but it is difficult to determine this IP in advance for the flatfile. So instead, pass a config file with the stack_service names and then change them to IP addresses using a script before launching H2O in each instance.

So, for example, use a docker-compose file that defines three services:

services:
  h2o_worker1:
    image: [h2o image]
    configs:
      - source: flatfile
        target: /flatfile
    deploy:
      placement:
        constraints:
          - node.hostname == [node1]
    ... 
  h2o_worker2:
    image: [h2o image]
    configs:
      - source: flatfile
        target: /flatfile
    deploy:
      placement:
        constraints:
          - node.hostname == [node1]
    ... 
  h2o_worker3:
    image: [h2o image]
    configs:
      - source: flatfile
        target: /flatfile
    deploy:
      placement:
        constraints:
          - node.hostname == [node1]
    ... 

##### Configs #####
configs:
  flatfile:
    file: flatfile

Where ... is other docker compose parameters you need to enter, and [] represents things you need to define for your setup.

Now create a flatfile based on the service names that will be imported by the config:

h2o_worker1:54321
h2o_worker2:54321
h2o_worker3:54321

Obviously, change the ports if necessary. Then use an entrypoint script to lookup each service name's IP, and then add 1 to get the $HOSTNAME IP for each service. I just use sleep here to make sure all the services have started so that the IP lookup works. Docker always appears to assign the two IPS per service sequentially, but YMMV. As I said, this is a hack and probably not a great production-level solution. My entrypoint script looks something like this:

echo "Moving flatfile to ${H2O_HOME}"
cp /flatfile ${H2O_HOME}

sleep 60
echo "Replacing hostnames in flatfile with IP addresses."
grep -o -P '.*(?=:)' ${H2O_HOME}/flatfile > ${H2O_HOME}/hostnames
grep -o -P '(?<=:).*' ${H2O_HOME}/flatfile > ${H2O_HOME}/ports
dig +short $(cat ${H2O_HOME}/hostnames) > ${H2O_HOME}/hostnames_ip
cat ${H2O_HOME}/hostnames_ip | awk -F"." '{printf "%d.%d.%d.%d\n", $1, $2, $3, $4 + 1}' > ${H2O_HOME}/new_ips
paste -d ":" ${H2O_HOME}/new_ips ${H2O_HOME}/ports > ${H2O_HOME}/new_flatfile

echo "Starting H2O..."
bash -c "java -Xmx${H2O_NODE_MEMORY:-1g} -jar ${H2O_HOME}/h2o.jar -flatfile ${H2O_HOME}/new_flatfile"

The key here is using dig to retrieve the IP addresses for each service host, and then incrementing by one to get the secondary address that we need to pass to H2O. Note I define an environment variable in my Dockerfile so I can vary the node memory in the docker compose file. You don't need to do that. And the Dockerfile also sets a variable for the install location for H2O, to simplify things.

This lets me deploy the containers using docker swarm, and H2O in fact finds all the nodes correctly. Because H2O does not permit additions or deletions of nodes after the initial setup, it is not a big deal (at least for me) to define most of this in advance. That said, I may yet try to move to weave or another network plugin that avoids some of these issues.