How can I replicate containers inside docker service when I restart docker daemon?

Question

I created docker service for the image percona XtraDB cluster with 3 replicas using the following command

docker service create \
--name mysql-galera \
--replicas 3 \
-p 3306:3306 \
--network mynet \
--env MYSQL_ROOT_PASSWORD=mypassword \
--env DISCOVERY_SERVICE=10.0.0.2:2379 \
--env XTRABACKUP_PASSWORD=mypassword \
--env CLUSTER_NAME=galera \
perconalab/percona-xtradb-cluster:5.6

I had already initialized docker swarm with three machines ( named with mach1, mach2, mach3) and all are joined as managers. And the replicas equally distributed to each of the three machines

When I tried to stop the docker daemon in mach2, docker created one more replica container in mach3. Again I restarted the docker daemon, mach3 was still running the two replicas and nothing on mach2. I manually removed the container in mach3 and mach2 was up with the 3rd replica

What should I do to automatically replicate containers on a restarted docker machine?

score 1 · Answer 1 · answered Apr 03 '18 at 06:19

1

I dont't think docker service re balance option is available in docker swarm ..... you can re distribute container by updating service with --force option

docker service update mysql-galera --force

answered Apr 03 '18 at 06:19

Ashok Reddy

1,060
1
16
28

This is correct. You want the moving of healthy workloads to be a ops-controlled operation. You should be designing your nodes so that one (or more, depending on your failure requirements) can fail and your performance/resources are still good. Once this is true, rushing to move workloads back to a different node won't be necessary. If you never want more then one replica per node, use global mode when creating the service. – Bret Fisher Apr 04 '18 at 00:43

score 0 · Answer 2 · answered Apr 03 '18 at 02:30

0

What you are looking for is a way to rebalance your replicaset automatically based on the availability of masters / slave nodes with capacity. However doing this would involve killing healthy containers and rescheduling them.

There is a open issue in docker swarm for this https://github.com/moby/moby/issues/24103

Right now if you want to do this the best way seems to be to scle up / down or create a rollout which will attempt to reschedule the containers in the swarm.

answered Apr 03 '18 at 02:30

Mohit Mutha

2,921
14
25

Can I achieve using the flag --restart-condition? Or will it solve half of this situation? – Manjunath PV Apr 03 '18 at 02:37
Not clear how are you proposing to use it but --restart-condition only defines a restart policy for the container. – Mohit Mutha Apr 03 '18 at 03:22

How can I replicate containers inside docker service when I restart docker daemon?

2 Answers2