1. Introduction
I've got Gitlab setup using a docker image https://github.com/sameersbn/docker-gitlab (sameersbn/docker-gitlab) and want to setup gitlab private registry.
1.1 Problem
All of this is behind reverse proxy load balancer using nginx. I am able to push into registry, but pull has some problem I am not able to resolve.
The output tells me retrying in 5sec and then increases over time, but after about minute it outputs Download complete on some layers. When all layers outputs Download complete. Pull results then like Unexpected EOF.
Infrastructure goes like this:
Proxy load balancer -----> gitlab registry
|
|
Another proxy load balancer -----> Gitlab
2. Terminal outputs
Docker login and push works fine. The problem is with pull.
2.1 Docker pull
2.1.1 expected result
root@server:~# docker pull registry.dalten.com/docker/php5.6-cli/image
Using default tag: latest
latest: Pulling from docker/php5.6-cli/image
10a267c67f42: Already exists
370377701f89: Pulling fs layer
455c73a122bc: Pulling fs layer
2cdcd82f5338: Pulling fs layer
3ac634f20449: Waiting
a19eeb6546ae: Waiting
4ca536f5a464: Waiting
4788559f39db: Waiting
cc1917e72a80: Waiting
42feb0ca362a: Pull complete
f74f29b91bbc: Pull complete
d104213bfd22: Pull complete
6db9d2d35d48: Pull complete
806b8ac2f0b7: Pull complete
661bb76178ac: Pull complete
3727be25d45f: Pull complete
04fed213644a: Pull complete
a78f18e044f7: Pull complete
849a2f576c66: Pull complete
259300decf26: Pull complete
0d9f42b121be: Pull complete
2483d273017a: Pull complete
3dfaf57e1d54: Pull complete
92fbf8438206: Pull complete
7d5100ec575c: Pull complete
52b1dff7320d: Pull complete
500fa7f84392: Pull complete
aff937c2773b: Pull complete
92f8bc1201f3: Pull complete
Digest: sha256:d3d06d2cebd333af21565639503ddded64db47d5641ee9ff93167200b5fcad7a
Status: Downloaded newer image for registry.dalten.com/docker/php5.6-cli/image:latest
2.1.2 Actual behavior
docker pull registry.dalten.com/docker/php5.6-cli/image
Using default tag: latest
latest: Pulling from docker/php5.6-cli/image
10a267c67f42: Already exists
370377701f89: Pulling fs layer
455c73a122bc: Pulling fs layer
2cdcd82f5338: Pulling fs layer
3ac634f20449: Waiting
a19eeb6546ae: Waiting
a19eeb6546ae: Downloading 4.398 MB/4.398 MB
4788559f39db: Downloading 102.6 MB/102.6 MB
cc1917e72a80: Download complete
42feb0ca362a: Download complete
f74f29b91bbc: Download complete
d104213bfd22: Download complete
6db9d2d35d48: Download complete
806b8ac2f0b7: Download complete
661bb76178ac: Download complete
3727be25d45f: Download complete
04fed213644a: Download complete
a78f18e044f7: Download complete
849a2f576c66: Download complete
259300decf26: Download complete
0d9f42b121be: Download complete
2483d273017a: Downloading 24.12 MB/24.12 MB
3dfaf57e1d54: Download complete
92fbf8438206: Download complete
7d5100ec575c: Download complete
52b1dff7320d: Download complete
500fa7f84392: Download complete
aff937c2773b: Download complete
92f8bc1201f3: Download complete
unexpected EOF
2.3 Docker login
sudo docker login registry.dalten.com
Username (xxxxxx):
Password:
Login Succeeded
2.4 Docker push
sudo docker push registry.dalten.com/docker/php5.6-cli/image
The push refers to a repository [registry.dalten.com/docker/php5.6-cli/image]
d8b353eb3025: Pushed
f2e85bc0b7b1: Pushed
fc9e1e5e38f7: Pushed
fe9a3f9c4559: Pushed
6a8bf8c8edbd: Pushed
latest: digest: sha256:ea1d854d38be82f54d39efe2c67000bed1b03348bcc2f3dc094f260855dff368 size: 1357
3. Configurations
Configurations contains docker-compose.yml files which I use for running services. Gitlab is running fine. Every configuration works. The only problem is with registry.
Another thing is that registry works while in insecured state. The problem must be hidden somewhere inside load balancer (nginx) configuration. (Some header has not been sent, limit exceeded, or so)
3.1 Gitlab Docker-compose
Note: I cutted compose of irrelevant information
version: '2'
services:
registry:
image: registry:latest
environment:
REGISTRY_AUTH_TOKEN_ISSUER: gitlab-issuer
REGISTRY_AUTH_TOKEN_REALM: https://git.dalten.cz/jwt/auth
REGISTRY_AUTH_TOKEN_SERVICE: container_registry
REGISTRY_LOG_LEVEL: debug
REGISTRY_STORAGE_DELETE_ENABLED: 'true'
REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE: /certs/fullchain.pem
REGISTRY_HTTP_SECRET: secret
volumes:
- /mnt/storage/srv/gitlab1/data/shared/registry:/registry
- /mnt/storage/srv/gitlab1/certs:/certs
ports:
- 5000:5000/tcp
gitlab:
image: sameersbn/gitlab:latest
hostname: git.dalten.cz
environment:
GITLAB_REGISTRY_ENABLED: 'true'
GITLAB_REGISTRY_HOST: registry.dalten.com
GITLAB_REGISTRY_API_URL: http://registry:5000
GITLAB_REGISTRY_ISSUER: gitlab-issuer
GITLAB_REGISTRY_CERT_PATH: /certs/fullchain.pem
GITLAB_REGISTRY_KEY_PATH: /certs/privkey.pem
volumes:
- /mnt/storage/srv/gitlab1/certs:/certs
- /mnt/storage/srv/gitlab1/data:/home/git/data/
ports:
- 8643:80/tcp
- 1022:22/tcp
3.2 Nginx registry docker-compose
version: '2'
services:
nginx:
image: sameersbn/nginx:1.10.3
volumes:
- /srv/nginx/sites-enabled:/etc/nginx/sites-enabled
- /mnt/storage/lb/letsencrypt:/etc/nginx/ssl
ports:
- 80:80/tcp
- 443:443/tcp
3.2.1 Nginx Virtual host file
I use Nginx for my reverse proxy configuration (Load Balancer). There are stored all certificates for https traffic.
# For versions of Nginx > 1.3.9 that include chunked transfer encoding
# support Replace with appropriate values where necessary
upstream docker-registry {
server 10.126.0.242:5000;
}
server {
listen 80;
server_name registry.dalten.com
return 301 https://$http_host:$request_uri;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
}
server {
listen 443 default_server;
server_name registry.dalten.com
add_header Docker-Distribution-Api-Version registry/2.0 always;
ssl on;
ssl_certificate /etc/nginx/ssl/live/registry.dalten.com/fullchain.pem;
ssl_certificate_key /etc/nginx/ssl/live/registry.dalten.com/privkey.pem;
ssl_session_timeout 5m;
client_max_body_size 0; # disable any limits to avoid HTTP 413 for large image
uploads
# required to avoid HTTP 411: see Issue #1486
# (https://github.com/dotcloud/docker/issues/1486)
chunked_transfer_encoding on;
proxy_set_header X-Original-URI $request_uri;
proxy_set_header Docker-Distribution-Api-Version registry/2.0;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
location /
{
# let Nginx know about our auth file
proxy_pass http://docker-registry;
proxy_set_header Host $host; # required for docker client's sake
proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
location /_ping {
proxy_pass http://docker-registry;
}
location /v1/_ping {
proxy_pass http://docker-registry;
}
location /v2/ {
# To add basic authentication to v2 use auth_basic setting plus
# add_header
add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always;
proxy_pass http://docker-registry;
proxy_set_header Host $http_host; # required for docker client's sake
proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 900;
}
}
3. Logs
If you require any more logs feel free to ask!
