1

I have created a two instance docker swarm on Google Compute Engine.

Docker version 18.06.1-ce, build e68fc7a on Ubuntu 18.04.1 LTS

I created a service account:

gcloud iam service-accounts create ${KEY_NAME} --display-name "${KEY_DISPLAY_NAME}"
gcloud projects add-iam-policy-binding $PROJECT --member serviceAccount:${KEY_NAME}@${PROJECT}.iam.gserviceaccount.com --role roles/storage.admin
gcloud iam service-accounts keys create --iam-account ${KEY_NAME}@${PROJECT}.iam.gserviceaccount.com key.json

Transferred the key.json to my docker swarm master:

Then I ran the following commands:

gcloud auth configure-docker
cat key.json | tr '\n' ' ' | docker login -u _json_key --password-stdin \
    https://eu.gcr.io

I can successfully pull an image from my private eu.gcr.io repository:

docker pull eu.gcr.io/$PROJECT/$IMAGE

So, logging in seems to work and the gcloud helper seems to be properly installed.

But creating a service in my swarm fails:

docker service create --replicas 2 --network overlay --name $NAME eu.gcr.io/$PROJECT/$IMAGE --with-registry-auth
image eu.gcr.io/$PROJECT/$IMAGE:latest could not be accessed on a registry to record
its digest. Each node will access eu.gcr.io/$PROJECT/$IMAGE:latest independently,
possibly leading to different nodes running different versions of the image.
qwdm524vggn50j4lzoe5paknj
overall progress: 0 out of 2 tasks 
1/2: No such image: eu.gcr.io/$PROJECT/$IMAGE:latest 
2/2: No such image: eu.gcr.io/$PROJECT/$IMAGE:latest

Looking in syslog shows the following:

Aug 25 13:37:15 mgr-1 dockerd[1368]: time="2018-08-25T13:37:15.299064551Z" level=info msg="Attempting next endpoint for pull after error: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication"
Aug 25 13:37:15 mgr-1 dockerd[1368]: time="2018-08-25T13:37:15.299168218Z" level=error msg="pulling image failed" error="unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication" module=node/agent/taskmanager node.id=xgozmc8iyjls7ulh4k3tvions service.id=qwdm524vggn50j4lzoe5paknj task.id=qrktpo34iuhiyl1rmbi71y4wg

AFAICS, I use the correct service account JSON to login into the Google Container Repository (as docker pull works), I added the flag --with-registry-auth to docker create service which has been the answer to similar questions, but still it doesn't work. Is docker create service working similar to docker pull?

Any ideas how I might solve this?

UPDATE

Instead of Google Container Registry I tried Gitlab Registry as well. Created a registry deploy token on the Gitlab site and entered the following commands:

docker login registry.gitlab.com -u $USERNAME -p $PASSWORD

Then this just works:

docker pull registry.gitlab.com/$ORGANISATION/$PROJECT/$IMAGE

But this command fails with a similar error:

docker service create --replicas 2 --network overlay  --name $NAME registry.gitlab.com/$ORGANISATION/$PROJECT/$IMAGE --with-registry-auth

image registry.gitlab.com/$ORGANISATION/$PROJECT/$IMAGE:latest could not be accessed on a registry to record
its digest. Each node will access registry.gitlab.com/$ORGANISATION/$PROJECT/$IMAGE:latest independently,
possibly leading to different nodes running different
versions of the image.
r5fqg94jrvt587le0fu779zaw
overall progress: 0 out of 2 tasks 
1/2: No such image: $ORGANISATION/$PROJECT/$IMAGE:latest
2/2: No such image: $ORGANISATION/$PROJECT/$IMAGE:latest

And /var/log/syslog contains

Aug 25 21:56:14 mgr-1 dockerd[1368]: time="2018-08-25T21:56:14.615895063Z" level=error msg="pulling image failed" error="Get https://registry.gitlab.com/v2/$ORGANISATION/$PROJECT/$IMAGE/manifests/latest: denied: access forbidden" module=node/agent/taskmanager node.id=xgozmc8iyjls7ulh4k3tvions service.id=r5fqg94jrvt587le0fu779zaw task.id=huwpjtu1wujk527t84y7yvbvd

So it seems docker create service doesn't use the credentials provided and the issue is not related to either Google Container Registry or Gitlab Registry?

Jeroen
  • 31
  • 1
  • 5

1 Answers1

2

OK, I found the problem. I had to use:

docker service create --with-registry-auth --replicas 2 --network overlay  --name $NAME registry.gitlab.com/$ORGANISATION/$PROJECT/$IMAGE

rather than

docker service create --replicas 2 --network overlay  --name $NAME registry.gitlab.com/$ORGANISATION/$PROJECT/$IMAGE --with-registry-auth

In the latter case the --with-registry-auth was considered an argument to my image rather than to the docker service create call and hence no authentication was used to pull the images from either private repository.

Jeroen
  • 31
  • 1
  • 5