Is there a way to limit a task to run on certain nodes, but not restrict to?

Question

I'm using a Docker Swarm which spans multiple data centers. The Swarm runs in a "virtual private cloud".

One of the data centers participating in the setup has a somewhat slower link than the others. I'd like to avoid a particularly latency sensitive service to be scheduled in that data center.

• I can use placement constraints, but that will impact availability. If the "slow" data center for some reason would be the only one running, constraints will ensure that the constrained service never gets scheduled in that data center. Impacts availability.
• I can use placement preferences, but that is just a preference. If I added the label lowlatency to all nodes except the ones in the "slow" data center, the service may still be scheduled there. It's after all just a preference.

Is there a way to somehow constrain a service to certain nodes, but allow Docker to schedule the service on other nodes if the preferred nodes are unavailable?

Answer 1

No, this is not a built-in feature with Swarm Mode.

The only placement preference currently available is to spread workload across values of a label. This is used to avoid all replicas being scheduled in a single availability zone, e.g. all workloads in the same rack, datacenter, etc. There is no placement preference that would act like a soft constraint.

The other available option in swarm scheduling is a constraint, and it's a hard restriction. Workloads will not be scheduled on nodes that do not match the constraint, even if that means they are not schedulable anywhere and the service remains down.

The closest you can come to your desired goal is to run an additional process to detect the outage of all the other datacenters and adjust the constraint on your service, however I suspect that will have two corresponding issues. First, with the other datacenters down, you have likely lost quorum with your managers, no scheduling activity will occur, and commands to any running manager will fail due to the loss of a leader. And second, if you do have quorum, the nodes in the remaining datacenter will likely become over-provisioned from the other workloads being rescheduled. That is known as the thundering herd issue which requires setting CPU and memory requirements on your containers. Those requirements would block new jobs from being scheduled on the node to avoid a further outage and prevent your altered service from finding a node with capacity.