docker push fails due to "unauthorized: authentication required", using gitlab

Question

I'm getting the following error when trying to push to the Gitlab registry using gitlab-runner:

unauthorized: authentication required 
ERROR: Build failed: exit status 1

Although:

$ docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN git.COMPANY.com
Login Succeeded

Doing build and push from my local env works just fine, suggesting the problem is related to the host which the runner runs on (gitlab-ci3), or maybe the user that is being used:

$ echo $USER
gitlab-runner

in group:

docker:x:999:gitlab-runner
gitlab-runner:x:998:

I've already tried docker unauthorized: authentication required - upon push with successful login but with no success. Maybe that gitlab-runner has no permission for root config.json is the cause?:

$ cat /root/.docker/config.json
 cat: /root/.docker/config.json: Permission denied

Beside solving this issue it would be very helpful if you would provide me with the how-to debug this error better for future times.

I'm using GitLab Enterprise Edition 8.13.1-ee, Docker 1.12.3, gitlab-ci-multi-runner 1.7.1

Gitlab output:

Running with gitlab-ci-multi-runner 1.7.1 (f896af7)
Using Shell executor...
Running on gitlab-ci3...
Fetching changes...
HEAD is now at cfe0a35 gitlab-ci.yml - testing
Checking out cfe0a356 as master...
$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 39
Server Version: 1.12.3
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 30
 Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: host null overlay bridge
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 3.13.0-96-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.612 GiB
Name: gitlab-ci3
ID: 6QWV:RQFD:4RWJ:D4CF:QN2M:MHKK:TABD:JD3F:3W7R:MCNA:4NHO:26VA
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8
$ docker-compose rm --all --force
The TESTS variable is not set. Defaulting to a blank string.
--all flag is obsolete. This is now the default behavior of `docker-compose rm`
No stopped containers
$ docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN git.COMPANY.com
Login Succeeded
$ docker build --pull -t $CONTAINER_IMAGE .
Sending build context to Docker daemon 557.1 kB
.
.
.
Sending build context to Docker daemon 18.05 MB


Step 1 : FROM ruby:latest
latest: Pulling from library/ruby
Digest: sha256:21a22bcc774f25c99bf5f27f16f8b4666f4ccbcb457e06f7a7e06a4728ea640b
Status: Image is up to date for ruby:latest
 ---> 45766fabe805
Step 2 : RUN apt-get update && curl -sL https://deb.nodesource.com/setup_5.x | bash - && apt-get install -y git nodejs && rm -rf /var/lib/apt/lists/*
 ---> Using cache
.
.
.
Step 23 : 
 ---> Using cache
 ---> b36460381a03
Successfully built b36460381a03
$ docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN git.COMPANY.com
Login Succeeded
$ echo $USER
gitlab-runner
$ cat ~/.docker/config.json
{
  "auths": {
    "git.COMPANY.com": {
      "auth": "EDW..."
    }
  }

 $ cat /root/.docker/config.json
 cat: /root/.docker/config.json: Permission denied

$ docker push $CONTAINER_IMAGE
The push refers to a repository [git.COMPANY.com:4567/ui/PROJECT]
fc4a22a92ee5: Preparing
.
.
.
149636c85012: Waiting
f96222d75c55: Waiting
unauthorized: authentication required
ERROR: Build failed: exit status 1

.gitlab-ci.yml:

before_script:
  - docker info
stages:
  - build
  - test
variables:
  CONTAINER_IMAGE: git.COMPANY.com:4567/ui/PROJECT:$CI_BUILD_REF_NAME
build:
  tags:
   - spec_work
  stage: build
  script:
      - docker-compose rm --all --force
      - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN git.COMPANY.com
      - docker build --pull -t $CONTAINER_IMAGE .
      - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN git.COMPANY.com
      - echo $USER
      - cat ~/.docker/config.json
      - docker push $CONTAINER_IMAGE
test:
  stage: test
  tags:
    - spec_work
  script:
    - docker-compose rm --all --force
    - docker-compose down
    - docker-compose build
    - docker-compose up --abort-on-container-exit --force-recreate
    - (exit `docker-compose ps | grep 'test' | grep -Po 'Exit (\d+)' | cut -d " " -f 2`)

Docker config.json:

root@gitlab-ci3 (Ubuntu 14.04) ➜  ~ cat .docker/config.json
{
  "auths": {
    "https://git.COMPANY.com": {
      "auth": "EDW..."
    }
  }
}

also tried with:

{
  "auths": {
    "https://git.COMPANY.com/ui/PROJECT": {
      "auth": "EDW..."
    }
  }
}

score 2 · Accepted Answer · answered Nov 02 '16 at 09:39

from Docker-in-Docker with Gitlab Shared runner for building and pushing docker images to registry

based on:

- docker version
- docker build -t $CI_REGISTRY_IMAGE:latest .
- docker tag $CI_REGISTRY_IMAGE:latest $CI_REGISTRY_IMAGE:$CI_BUILD_TAG
- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY
- docker push $CI_REGISTRY_IMAGE:$CI_BUILD_TAG

I've used:

- docker version
- docker build -t $CI_REGISTRY_IMAGE:latest .
- docker tag $CI_REGISTRY_IMAGE:latest $CI_REGISTRY_IMAGE:$CI_BUILD_REF_NAME
- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY
- docker push $CI_REGISTRY_IMAGE:$CI_BUILD_REF_NAME

in build section at .gitlab-ci.ym.

Now the push works just fine.

As to WHY it works, I don't know. If anybody knows why this change resolved this issue please comment, I do wish to know why.

thanks!

score 0 · Answer 2 · answered Oct 31 '16 at 12:31

0

Please check if your environment variable http_proxy and https_proxy are different from blank. Also check the default docker configuration file for the equivalent parameters.

Regards

answered Oct 31 '16 at 12:31

Carlos Rafael Ramirez

5,984
1
29
36

there is no env variable related to http/s_proxy, by default docker do you mean /etc/default/docker? if so there is no proxy var set there. – Captain_Meow_Meow Nov 01 '16 at 06:23

score 0 · Answer 3 · answered Feb 20 '19 at 10:17

The root cause:

correct:

docker login ... git.COMPANY.com:4567
docker push      git.COMPANY.com:4567/something

correct:

docker login ... git.COMPANY.com
docker push      git.COMPANY.com/something

incorrect:

docker login ... git.COMPANY.com
docker push      git.COMPANY.com:4567/something

Different port means a different registry. The incorrect login that you initially used didn't store credentials usable for the push.

docker push fails due to "unauthorized: authentication required", using gitlab

3 Answers3