Docker: Swarm worker nodes not finding locally built image

Question

Maybe I missed something, but I made a local docker image. I have a 3 node swarm up and running. Two workers and one manager. I use labels as a constraint. When I launch a service to one of the workers via the constraint it works perfectly if that image is public.

That is, if I do:

docker service create --name redis --network my-network  --constraint node.labels.myconstraint==true redis:3.0.7-alpine

Then the redis service is sent to one of the worker nodes and is fully functional. Likewise, if I run my locally built image WITHOUT the constraint, since my manager is also a worker, it gets scheduled to the manager and runs perfectly well. However, when I add the constraint it fails on the worker node, from docker service ps 2l30ib72y65h I see:

... Shutdown       Rejected 14 seconds ago  "No such image: my-customized-image"

Is there a way to make the workers have access to the local images on the manager node of the swarm? Does it use a specific port that might not be open? If not, what am I supposed to do - run a local repository?

BMitch · Accepted Answer · 2018-08-11T11:50:53.823

14

The manager node doesn't share out the local images from itself. You need to spin up a registry server (or user hub.docker.com). The effort needed for that isn't very significant:

# first create a user, updating $user for your environment:
if [ ! -d "auth" ]; then
  mkdir -p auth
fi
touch auth/htpasswd
chmod 666 auth/htpasswd
docker run --rm -it \
  -v `pwd`/auth:/auth \
  --entrypoint htpasswd registry:2 -B /auth/htpasswd $user
chmod 444 auth/htpasswd

# then spin up the registry service listening on port 5000
docker run -d -p 5000:5000 --restart=always --name registry \
  -v `pwd`/auth/htpasswd:/auth/htpasswd:ro \
  -v `pwd`/registry:/var/lib/registry \
  -e "REGISTRY_AUTH=htpasswd" \
  -e "REGISTRY_AUTH_HTPASSWD_REALM=Local Registry" \
  -e "REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd" \
  -e "REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY=/var/lib/registry" \
  registry:2

# then push your image
docker login localhost:5000
docker tag my-customized-image localhost:5000/my-customized-image
docker push localhost:5000/my-customized-image

# then spin up the service with the new image name
# replace registryhost with ip/hostname of your registry Docker host
docker service create --name custom --network my-network \
  --constraint node.labels.myconstraint==true --with-registry-auth \
  registryhost:5000/my-customized-image

edited Aug 11 '18 at 11:50

answered Sep 07 '16 at 14:00

BMitch

231,797
42
475
450

Sweet - thank you! Before I accept as answer though, (I don't like inconsistency!), doesn't it seem inconsistent that the Manager node DOES see the local images (not from docker run, but from docker service create) and the Worker nodes do not? – JoeG Sep 07 '16 at 14:24
From the user perspective, maybe. But from the code perspective, each Docker host is a separate environment. When you do the build, you are creating that image on the host, not on the swarm. The logic on the `service create` is to first use what's cached on the host, and then fall back to pulling that image from the registry. Which happens to be an identical behavior to `docker run`. – BMitch Sep 07 '16 at 15:05
I may be missing something, but how does the worker node in the swarm refer to this registry/image? I tried putting localhost:5000/my/image in my yml file, but my worker nodes still cannot find the image. Is there some way to make this work without having to log into the worker nodes? – Eric Kolotyluk Aug 28 '17 at 17:15
@EricKolotyluk `registryhost:5000/my-customized-image`, localhost will be the host of the worker, you need to point them to the registry server hostname. – BMitch Aug 28 '17 at 17:18
For example, I have started the registry as above, which is running on 172.17.0.2. In my yml file I have `image: 172.17.0.2:5000/iggcanada/kafka` but when I do a `docker swarm deploy -c kafka-stack.yml kafka` the output from `docker stack ps kafka` shows `No such image: 172.17.0.2:500…` – Eric Kolotyluk Aug 28 '17 at 17:36
I missed an important step covered by elkoo in his answer (I'm used to making secure registries with a local CA that's trusted by all my hosts). You need to configure the workers to accept your host as an insecure registry. If you're following the advice here and still have problems, please consider opening up a new question. – BMitch Aug 28 '17 at 17:41
Okay, finally had some success. I am able to get things working with the AWS registry using `docker stack deploy --with-registry-auth -c kafka-stack.yml kafka` where I refer to the image via the AWS registry. For some reason I cannot refer to my swarm registry at `172.17.0.2:5000/iggcanada/kafka` – Eric Kolotyluk Aug 28 '17 at 17:55
Basically I am trying to avoid having anything administrative to do on my worker nodes. This swarm was created on AWS via the Docker Swarm CloudFormation template, and they have set things up to make it hard to do anything on the worker nodes directly. – Eric Kolotyluk Aug 28 '17 at 17:57
@erickolotyluk if you're in AWS, it could be your security policy not allowing connections to that port. – BMitch Aug 28 '17 at 17:57
In the end, I was able to connect to Kafka on my worker nodes via the ELB that the Docker CloudFormation template sets up, but because Kafka is configured differently than Zookeeper in the compose.yml file, an ELB listener was not set up for Kafka. Getting configuration correct is hell. – Eric Kolotyluk Sep 08 '17 at 19:50
Creating a user, 'chmod 666 auth/htpasswd': how does one get the file 'htpasswd'? – Nir Aug 11 '18 at 11:49
1

@Nir good catch. Might work if the file doesn't exist, but I've added a touch command to create it. – BMitch Aug 11 '18 at 11:52

score 2 · Answer 2 · edited Oct 21 '18 at 11:47

For me, this step-by-step guide worked. However, it is insecure:

# Start your registry
$ docker run -d -p 5000:5000 --name registry registry:2

# Tag the image so that it points to your registry
$ docker tag my_existing_image localhost:5000/myfirstimage

# Push it to local registry/repo
$ docker push localhost:5000/myfirstimage

# For verification you can use this command:
$ curl -X GET http://localhost:5000/v2/_catalog
# It will print out all images on repo.

# On private registry machine add additional parameters to enable insecure repo:
ExecStart=/usr/bin/dockerd --insecure-registry IP_OF_CURRENT_MACHINE:5000

# Flush changes and restart Docker:
$ systemctl daemon-reload
$ systemctl restart docker.service

# On client machine we should say docker that this private repo is insecure, so create or modifile the file '/etc/docker/daemon.json':
{ "insecure-registries":["hostname:5000"] }

# Restart docker:
$ systemctl restart docker.service

# On swarm mode, you need to point to that registry, so use host name instead, for example: hostname:5000/myfirstimage

I may be missing something, but how does the worker node in the swarm refer to this registry/image? I tried putting `localhost:5000/my/image` in my yml file, but my worker nodes still cannot find the image. Is there some way to make this work without having to log into the work nodes? — Eric Kolotyluk, Aug 28 '17 at 16:36

score 0 · Answer 3 · answered Sep 07 '16 at 14:00

Images have to be downloaded to the local cache on each node. The reason is that if you store all of your images on one node only and that node goes down, swarm would have no way to spawn new tasks (containers) on the other nodes.

I personally just pull a copy of all the images on each node before starting the services. That can be done in a bash script or Makefile (eg below)

pull:
  @for node in $$NODE_LIST; do
    OPTS=$$(docker-machine config $$node)
    set -x
    docker $$OPTS pull postgres:9.5.2
    docker $$OPTS pull elasticsearch:2.3.3
    docker $$OPTS pull schickling/beanstalkd
    docker $$OPTS pull gliderlabs/logspout
    etc ...
    set +x
done

I believe the OP is looking for a solution that covers locally built images, not those from hub.docker.com. — BMitch, Sep 07 '16 at 14:05
@BMitch You're right. I didn't read the question correctly. Time to hit the bed here. — Bernard, Sep 07 '16 at 14:09

Docker: Swarm worker nodes not finding locally built image

3 Answers3

Linked