0

docker service create ... works even though compressed image size is 0B in Docker Hub. On the other hand, when I use Artifactory as private registry, it fails with No such image error. Docker daemons' debug logs say manifest verification failed for digest ...

As an example, compressed size of portainer's latest tag and main release tags (1.13.1, 1.13.2, etc.) are 0 B: https://hub.docker.com/r/portainer/portainer/tags/

Following command works:

docker service create \
  --name portainer \
  --publish 9000:9000 \
  --constraint 'node.role == manager' \
  --mount type=bind,src=//var/run/docker.sock,dst=/var/run/docker.sock \
  portainer/portainer \
  -H unix:///var/run/docker.sock

but following command does not work:

docker service create \
  --name portainer \
  --publish 9000:9000 \
  --constraint 'node.role == manager' \
  --mount type=bind,src=//var/run/docker.sock,dst=/var/run/docker.sock \
  artifactory.mycompany.com/portainer/portainer \
  -H unix:///var/run/docker.sock

Service's state:

[myuser@rose1]$ docker service ps --no-trunc  portainer
ID                          NAME                IMAGE                                                                                                                NODE                DESIRED STATE       CURRENT STATE             ERROR                                                                                                                                 PORTS
gzk05p5x89w9pcvenuyio8pu8   portainer.1         artifactory.mycompany.com/portainer/portainer:latest@sha256:5393dc7fc9e93f8ca8b034941a2c6af0ae176c89c92728d4ff0e110d0558cd40   rose1             Ready               Rejected 2 seconds ago    "No such image: artifactory.mycompany.com/portainer/portainer:latest@sha256:5393dc7fc9e93f8ca8b034941a2c6af0ae176c89c92728d4ff0e110d0558cd40"
fcovqtudbv3zmgo4von01y5wv    \_ portainer.1     artifactory.mycompany.com/portainer/portainer:latest@sha256:5393dc7fc9e93f8ca8b034941a2c6af0ae176c89c92728d4ff0e110d0558cd40   rose1             Shutdown            Rejected 7 seconds ago    "No such image: artifactory.mycompany.com/portainer/portainer:latest@sha256:5393dc7fc9e93f8ca8b034941a2c6af0ae176c89c92728d4ff0e110d0558cd40"
jfy8lr2prypcx72dryse5vmwx    \_ portainer.1     artifactory.mycompany.com/portainer/portainer:latest@sha256:5393dc7fc9e93f8ca8b034941a2c6af0ae176c89c92728d4ff0e110d0558cd40   rose1             Shutdown            Rejected 12 seconds ago   "No such image: artifactory.mycompany.com/portainer/portainer:latest@sha256:5393dc7fc9e93f8ca8b034941a2c6af0ae176c89c92728d4ff0e110d0558cd40"
3ovw7pwgr6srhvqocrqayiuqx    \_ portainer.1     artifactory.mycompany.com/portainer/portainer:latest@sha256:5393dc7fc9e93f8ca8b034941a2c6af0ae176c89c92728d4ff0e110d0558cd40   rose1             Shutdown            Rejected 12 seconds ago   "No such image: artifactory.mycompany.com/portainer/portainer:latest@sha256:5393dc7fc9e93f8ca8b034941a2c6af0ae176c89c92728d4ff0e110d0558cd40"

Docker daemon's debug logs:

...
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.137611299+03:00" level=debug msg="Trying to pull artifactory.mycompany.com/portainer/portainer from https://artifactory.mycompany.com v2"
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.169441596+03:00" level=debug msg="task status updated" method="(*Dispatcher).processUpdates" module=dispatcher node.id=xdn6m020ugsnbfqfk2
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.169573572+03:00" level=debug msg="task status updated" method="(*Dispatcher).processUpdates" module=dispatcher node.id=xdn6m020ugsnbfqfk2
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.175689648+03:00" level=debug msg="Pulling ref from V2 registry: artifactory.mycompany.com/portainer/portainer:latest@sha256:5393dc7fc9e93f8ca8b0349
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.175757143+03:00" level=error msg="manifest verification failed for digest sha256:5393dc7fc9e93f8ca8b034941a2c6af0ae176c89c92728d4ff0e110d
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.175783178+03:00" level=info msg="Attempting next endpoint for pull after error: manifest verification failed for digest sha256:5393dc7fc9
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.175800969+03:00" level=debug msg="Skipping v1 endpoint https://artifactory.mycompany.com because v2 registry was detected"
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.175878617+03:00" level=debug msg="pull in progress"
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.175909141+03:00" level=error msg="pulling image failed" error="manifest verification failed for digest sha256:5393dc7fc9e93f8ca8b034941a2
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.176596565+03:00" level=error msg="fatal task error" error="No such image: artifactory.mycompany.com/portainer/portainer:latest@sha256:5393dc7fc9e93
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.176643801+03:00" level=debug msg="state changed" module="node/agent/taskmanager" node.id=xdn6m020ugsnbfqfk2f5g74jx service.id=ve3ipsb1cx3
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.176882355+03:00" level=debug msg="(*Agent).UpdateTaskStatus" module="node/agent" node.id=xdn6m020ugsnbfqfk2f5g74jx task.id=3rzww5i46b8sv3
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.177387272+03:00" level=debug msg="task status reported" module="node/agent" node.id=xdn6m020ugsnbfqfk2f5g74jx
...

Artifactory logs:

...
2017-06-09 14:00:11,725 [http-nio-8081-exec-1] [INFO ] (o.a.a.d.r.v.r.v.DockerV2VirtualRepoHandler:105) - Fetching docker manifest for repo 'portainer/portainer' and tag 'latest'
2017-06-09 14:00:14,940 [http-nio-8081-exec-1] [INFO ] (o.a.r.HttpRepo      :420) - registry-1.docker.io downloading https://registry-1.docker.io/v2/portainer/portainer/manifests/latest 944 bytes
2017-06-09 14:00:14,948 [http-nio-8081-exec-1] [INFO ] (o.a.r.HttpRepo      :433) - registry-1.docker.io downloaded  https://registry-1.docker.io/v2/portainer/portainer/manifests/latest 944 bytes at 125.43 KB/sec
2017-06-09 14:00:15,194 [http-nio-8081-exec-5] [INFO ] (o.a.a.d.r.v.r.v.DockerV2VirtualRepoHandler:105) - Fetching docker manifest for repo 'portainer/portainer' and tag 'latest'
2017-06-09 14:00:15,529 [http-nio-8081-exec-7] [INFO ] (o.a.a.d.r.v.r.v.DockerV2VirtualRepoHandler:105) - Fetching docker manifest for repo 'portainer/portainer' and tag 'latest'
2017-06-09 14:00:20,526 [http-nio-8081-exec-8] [INFO ] (o.a.a.d.r.v.r.v.DockerV2VirtualRepoHandler:105) - Fetching docker manifest for repo 'portainer/portainer' and tag 'latest'
...

Update 1:

docker pull ... works properly:

docker pull artifactory.mycompany.com/portainer/portainer

and docker run ... also works properly:

docker run \ 
  -v /var/lib/docker.sock:/var/lib/docker.sock \
  -p 9000:9000 \
  artifactory.mycompany.com/portainer/portainer \
  -H unix:///var/run/docker.sock

The problem only exists with swarm mode.

Update 2:

As @Tony pointed out, if image is a multi-arch manifest (hence the 0B size) I have issues with Artifactory. For example, all the images under https://hub.docker.com/u/trollin are multi-arch and each tag of each image seem 0 Byte. I can reproduce the same issue with these images & tags. Take trollin/nginx, as an example.

Following commands work:

1)

docker pull artifactory.mycompany/trollin/nginx

2)

docker run --name trollin_nginx \
  --publish 9991:80 \
  artifactory.mycompany/trollin/nginx

3)

docker service create \
  --name trollin_nginx \
  --publish 9991:80 \
  trollin/nginx

Following command does not work:

docker service create \
  --name trollin_nginx \
  --publish 9991:80 \
  artifactory.mycompany.com/trollin/nginx
Ali Sadik Kumlali
  • 631
  • 1
  • 6
  • 14
  • 1
    Can you run `docker run -v /var/lib/docker.sock:/var/lib/docker.sock -p 9000:9000 artifactory.mycompany.com/portainer/portainer -H unix:///var/run/docker.sock`? If not, then the problem isn't with swarm and I'd remove that from your debugging steps. – BMitch Jun 09 '17 at 14:00
  • It works. The problem only exists with swarm mode. I added Update 1. – Ali Sadik Kumlali Jun 12 '17 at 06:27
  • Does `docker service create --with-registry-auth ...` make any difference? – BMitch Jun 12 '17 at 08:06
  • Did not work either. – Ali Sadik Kumlali Jun 13 '17 at 13:59
  • In that case, it may be a swarm mode defect. Have a look a the swarmkit issue list to see if anything matches, and if not, open a new issue. – BMitch Jun 13 '17 at 14:07
  • 1
    Have you tried with another image? Something simpler like nginx for example? Portainer image is a multi-arch manifest (hence the 0B size), I'm not sure how you managed to push it onto your own registry. – Tony Jun 14 '17 at 07:35
  • @BMitch If it was a swarm mode defect, the same problem would happen with Docker Hub, as well. But the problem only exists with Artifactory. I am not sure whether it is a Artifactory bug or a configuration issue. What do you think? – Ali Sadik Kumlali Jun 20 '17 at 09:02
  • Thanks @Tony! You let me point out the core issue. I have added Update 2. For your second part of the comment, I did not push `portainer/portainer` image to Artifactory. When I pull it through Artifactory (artifactory.mycompany.com/portainer/portainer), it somehow pull the image from Docker Hub and store it. I do not know how Artifactory handles multi-arch manifest. But, it seems there is a problem with it, at least while creating swarm services. – Ali Sadik Kumlali Jun 20 '17 at 10:26
  • No worries, you might want to report that to the Artifactory project then ;) – Tony Jun 20 '17 at 14:48

0 Answers0