0

I'm using docker swarm, and storing images in AWS ECR. I'm using auto-scaling, so instances go up and down all the time. When nodes start, they join the cluster and docker swarm deploys the container.

That works fine for a few hours after I create the service, then at some point docker stops deploying containers to the nodes until I manually go there and run docker swarm update --with-registry-auth myservice. It deploys all the containers and works for a few more hours, then stops again.

I'm using aws ecr credential helper to login to the registry, and it has been working fine. But my guess is that docker is not updating the credentials automatically form time to time, as each token are only valid for a few hours.

What do I need to do to make docker swarm update the credentials automatically from time to time? I suppose I could run docker service update --with-registry-auth on cron every hour, but I'm not sure if this is the right path.

Natan
  • 4,686
  • 5
  • 30
  • 48
  • Would that even work? I guess you'd have to automate the AWS login to get the new docker auth, then pipe those to a new `docker login`, then do a service update, but you'd need to test that last bit. This is why I don't use ECR with Docker engine or Swarm's that I build myself... :) – Bret Fisher Jan 27 '18 at 03:06
  • The aws login is already automated with the ecr helper. – Natan Jan 27 '18 at 08:05
  • I recommend putting this in https://github.com/moby/moby/issues as an issue. A good title might be "docker swarm doesn't update services with new engine auth from AWS ECR helper". Really, I think the core issue is that we don't have a working or documented way to auto-update a swarm service with new credentials. ECR helper might solve it for the local engine, but I bet it doesn't understand that swarm stores creds per-service (I assume). – Bret Fisher Jan 27 '18 at 22:18

1 Answers1

0

I was working on this answer and saw your comment on one of the key issues tracking this, so figured you're already farther along then me.

Not being an ECR person much, I needed to catch up :)

This ECR helper issue and the moby/moby issues it mentions (and their nested issues) I'm understanding the auth storage and limitation in Swarm service definitions a bit better.

Yes, without the proper native Docker support, I bet the final solution would be something like you suggest. I don't like doing anything in a Swarm unless it's a Swarm service, so:

  1. Get ECR helper working in a container and create a Service that bind-mounts a manager nodes docker socket.
  2. It performs auth on the local host engine.
  3. You have a cron container, as a 2nd Service, also constrained to a manager, that docker service update --with-registry-auth for each service using ECR. I assume this doesn't cause task re-create, but haven't tested it.

Then I found this comment and it sounds like someone's done just that. However, it seems the repo only works for swarms created with the Docker for AWS CloudFormation template. It would be great if the solution would also work for any swarm in AWS.

If you get a working setup and create a gist or repo about it, I'm happy to help or review.

Bret Fisher
  • 8,164
  • 2
  • 31
  • 36
  • I'm using ECR for convenience only. I tried the crontab solution, running `docker service ls | tail -n +2 | awk '{print $2}' | xargs -L 1 docker service update --with-registry-auth -d`, but for some reason it didn't work. Maybe when cron runs, it doesn't run exactly as an user. This morning, a lot of stopped services. Running exactly the same command manually fixes it. I guess I'll simply get out of ECR and store the images either on docker hub or setup my own repo. Everyting ECS-related has been a disaster for me so far. – Natan Jan 29 '18 at 11:38