I am having issues with swarm worker nodes not updating images when doing either an update or deploy (on an existing stack). The stack will always work when created
This solution only works on creation.
To reproduce the issue, do the following
1) create a container, something like httpd with an index.html, store it at private-registry.example.com/path/image
2) create test.yml
version: '3.4'
services:
test:
# Use the build in the current pipeline
image: private-registry.example.com/path/image
deploy:
replicas: 3
3) deploy stack
docker login private-registry.example.com
docker stack deploy --with-registry-auth --compose-file=test.yml test
4) update the container, change some text
5) re-deploy the stack
docker login private-registry.example.com
docker stack deploy --with-registry-auth --compose-file=test.yml test
the swarm manager will have the latest image, the swarm nodes will not.
docker service ps test_test
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
j9497jwolee4 test_test.1 private-registry.example.com/path/image swarm-slave-01.example.com Running Running 5 seconds ago
zsqxx3m0mpk3 \_ test_test.1 private-registry.example.com/path/image swarm-slave-01.example.com Shutdown Shutdown 7 seconds ago
sjjggcqmjcvo test_test.2 private-registry.example.com/path/image swarm-master.example.com Running Running 10 seconds ago
uyey60wv2vsc \_ test_test.2 private-registry.example.com/path/image swarm-slave-01.example.com Shutdown Rejected 20 seconds ago "No such image: private-registry..."
ttzvf4j3whk3 \_ test_test.2 private-registry.example.com/path/image swarm-slave-01.example.com Shutdown Rejected 25 seconds ago "No such image: private-registry..."
x77e3r46zl1j \_ test_test.2 private-registry.example.com/path/image swarm-master.example.com Shutdown Rejected 31 seconds ago "No such image: private-registry..."
5a7lywn6zycz \_ test_test.2 private-registry.example.com/path/image swarm-master.example.com Shutdown Rejected 36 seconds ago "No such image: private-registry..."
qp1acqgthl33 test_test.3 private-registry.example.com/path/image swarm-slave-02.example.com Running Running 11 seconds ago
osyn19o6c30j \_ test_test.3 private-registry.example.com/path/image swarm-master.example.com Shutdown Shutdown 12 seconds ago
WORKAROUND This pulls the lastest image everytime without issue.
docker login private-registry.example.com
docker stack rm test
docker stack deploy --with-registry-auth --compose-file=test.yml test
System
Server Version: 18.06.1-ce
Operating System: Ubuntu 18.04.1 LTS
