A simplified version of my current setup is as follows:
- 3 virtual machines: VM1, VM2, VM3
- Each of them runs a
docker-compose
stack that includes a tomcat app and a DB.
The DB is recreated nightly from raw data (running a docker container that contains an indexing app).
This is achieved with crontab jobs, that are scheduled so that only 1 VM is in maintenance at any given time.
Now, this is going to scale up to several more VMs and possibly multiple replicas of this stack per VM. I am looking into docker swarm stacks to simplify the deployment and management.
The specific question is:
Once the deployment is managed via swarm, especially if the service is replicated into more than 1 instance per VM, how can I manage the nightly DB recreation, so that:
- all DBs are recreated
- I can control how many service instances, out of my cluster, are in maintenance mode at any given time?
I guess I could reformulate the question as: how to control maintenance at service level (vs node level)?
I do know that swarm nodes can be drained independently and be put in maintenance mode. The problem is that each node can contain a number of replicas of a service, and each of these replicas needs its own maintenance.