44

I would like to know how to configure this tool to start with a minimum number of nodes and grow to a maximum number of nodes when it is required.

Should I use another tool or can docker swarm solve this for me?

Celso Agra
  • 1,389
  • 2
  • 15
  • 37
  • According to a Proposal/Feature Request in the [official Swarm repo](https://github.com/docker/swarmkit/issues/1663), the [dockercloud-haproxy](https://github.com/docker/dockercloud-haproxy#running-with-docker-swarmmode-swarm-mode) tool looks like it works with Swarm out of the box, but I have not confirmed. – smeeb Jun 14 '17 at 02:02
  • @smeeb does that tool include autoscaling? – 030 Jul 23 '17 at 11:33
  • dockercloud-haproxy distributes requests along existing containers, it does not aid in auto-scaling. – raarts Dec 05 '17 at 11:03

2 Answers2

47

Short answer: There is no easy way to do this with Docker Swarm for now.

Docker Swarm (or Swarm mode) does not support auto-scaling machines out of the box. You'd need to use another solution for that like docker-machine to create machines (with docker) on your infrastructure and link these to the existing Swarm cluster (with docker swarm join).

This will involve a lot of scripting but the idea is to monitor the cluster for CPU / Memory / Network usage (with top or monit) and once it goes beyond a threshold (say 70% of total cluster resources), you trigger a script calling docker-machine to scale up the cluster. Using the same idea you can also scale down by draining and removing nodes (preferably Agent nodes) from the existing swarm cluster once your are below the lower threshold.

You need to make sure you are monitoring for sustained resource usage if you want to use this criteria or you will have your Infrastructure spawning and destroying nodes from the frequent and sudden changes in resource usage.

You can define a lower bound and an upper bound for machines in the cluster to keep things under control.

Note that Swarm requires at least 3 Manager nodes (recommended 5) to maintain a quorum for the Distributed Consensus algorithm. So the minimum recommended lower bound is 5 nodes (which you can extend with Agent nodes as resources are incrementally being used by services).

To some extent, you can also take a look at Docker InfraKit or Terraform for Infrastructure automation and Health monitoring.

Update: There is now a promising cross-platform autoscaler that supports Swarm Mode task auto-scaling: Orbiter. Although still nothing out-of-the-box yet for service/machine autoscaling.

abronan
  • 3,309
  • 4
  • 29
  • 38
  • Thanks @abronan! As I understand, nodes can be scalable only with external tools. What about containers? would be possible to autoscale containers? Or should I have use another tool such as kubernetes? – Celso Agra Jan 16 '17 at 13:15
  • 2
    @CelsoAgra AFAIK, there is the `docker service scale` command (or its `docker-compose` counterpart) but you also have to trigger the scaling using stats on incoming requests or cpu usage for example. Kubernetes indeed has a [pod autoscaler](https://kubernetes.io/docs/user-guide/horizontal-pod-autoscaling/) based on resource usage so this can fit your use case :) – abronan Jan 16 '17 at 13:42
  • @abronan (or anyone) do you know if the Docker community has a built-in auto-scaler planned on the Docker Roadmap? I feel like this would be a heavily-utilized feature if Docker supported this right out of the box... – smeeb Jun 14 '17 at 01:53
  • Also @abronan any chance you can explain: (1) what component is typically monitoring cluster-wide resources and, most importantly, (2) what component runs the `docker-machine` commands? – smeeb Jun 14 '17 at 01:55
  • 3
    We needed autoscaling based on metrics stored in prometheus. We built a service which uses docker API to scale based on metrics from prometheus https://github.com/sahajsoft/docker-swarm-service-autoscaler – Deepak N Jan 11 '18 at 14:34
3

Scaling up is pretty easy. You can just keep on scheduling containers when needed. Then you just create a script that looks for pending container and scales up the cluster. For example if you're using the official CloudFormation template on aws for swarm, you can just change the desired number on the autoscaling group. An example iteration script could looks like this:

services=$(docker service ls --format '{{.ID}}')
for service in $services; do
  tasks=$(docker service ps $service --format '{{.ID}}')
  for task in $tasks; do
    if docker inspect $task --format '{{.Status}}' | grep 'insufficient resources' 1>/dev/null; then
      scale-up-cmd
    fi
  done
done
aaron
  • 1,176
  • 10
  • 7