2

I'm trying to implement a backup system. This system requires the execution of a docker container at the time of the backup on the specific node. Unfortunately I have not been able to get it to execute on the required node.

This is the command I'm using and executing on the docker swarm manager node. It is creating the container on the swarm manager node and not the one specified in the constraint. What am I missing?

docker run --rm -it --network cluster_02 -v "/var/lib:/srv/toBackup" \
   -e BACKUPS_TO_KEEP="5" \
   -e S3_BUCKET="backup" \
   -e S3_ACCESS_KEY="" \
   -e S3_SECRET_KEY="" \
   -e constraint:node.hostname==storageBeta \
   comp/backup create $BACKUP_NAME
user2693017
  • 1,750
  • 6
  • 24
  • 50

1 Answers1

2

You are using an older classic Swarm method to try running your container, but almost certainly using Swarm Mode. If you installed your swarm with docker swarm init and can see nodes with docker node ls on the manager, then this is the case.

Classic Swarm ran as a container that was effectively a reverse proxy to multiple docker engines, intercepting calls to commands like docker run and sending them to the appropriate node. It is generally recommended to avoid this older swarm implementation unless you have a specific use case and take the time to configure mTLS on each of your docker hosts.

Swarm Mode provides an HA manager using Raft for quorum (same as etcd), handles encryption of all management requests, configures overlay networking, and works with a declarative model, giving the target state, rather than imperative commands to run. It's a very different model from classic Swarm. Notably, Swarm Mode only works on services and stacks, and completely ignores the docker run and other local commands, e.g.:

docker service create \
  --name backup \
  --constraint node.hostname==storageBeta \
  --network cluster_02 \
  -v "/var/lib:/srv/toBackup" \
  -e BACKUPS_TO_KEEP="5" \
  -e S3_BUCKET="backup" \
  -e S3_ACCESS_KEY="" \
  -e S3_SECRET_KEY="" \
  comp/backup create $BACKUP_NAME

Note that jobs are not well supported in Swarm Mode yet (there is an open issue seeking community feedback on including this functionality). It is currently focused on running persistent services that do not normally exit unless there is an error. If your command is expected to exit, you can include an option like --restart-max-attempts 0 to prevent swarm mode from restarting it. You may also have additional work to do if the network is not swarm scoped.

I'd also recommend converting this to a docker-compose.yml file and deploying with docker stack deploy to better document the service as a config file.

BMitch
  • 231,797
  • 42
  • 475
  • 450
  • Thank you for the very thorough explanation. We are using swarm for the cluster already and now want to initiate the backups from the swarm manager. I see if using --restart-max-attempts works well for us. Do you know if it throws a proper error/a workaround for when it fails so that we can something like `/home/user/backup.sh && curl -fsS --retry 3 https://hc-ping.com/62241e9e-a4aa-40e6-8ad1-15f21d884f87 > /dev/null` – user2693017 Nov 09 '19 at 12:42
  • @user2693017 that's something you'll want to test in your environment. I typically avoid restart policies since I'm running services rather than jobs, and container restart policies can conflict with the swarm scheduler in that scenario. – BMitch Nov 09 '19 at 13:18
  • I see. Is there anything you would recommend? I could open a shell on the remote server to do this but would like to avoid that if possible. – user2693017 Nov 10 '19 at 16:38
  • 1
    @user2693017 by test in your environment, I'm thinking you could create various services with different restart policy flags to see if you can get the desired behavior. The service could intentionally fail for you to test your error handling. – BMitch Nov 11 '19 at 18:42