Gitlab Container Registry timeout when connecting

Question

Building an on-prem / self-hosted deployment of Gitlab for the first time. I've got the self-installation completed, single box and three dedicated hosts for runners (that I'm working to get them dockerized for docker-in-docker).

Did this from a package install two days ago, so nothing built from source, 99% of the things are still from out-of-the-box omnibus installer.

I'm trying to get the Container Registry up and running so that I can check-in containers and pull them for my CI/CD pipeline, but every time my local host tries to authenticate, it times out. When I enable the debug logs, it doesn't seem to generate anything?

PS C:\Users\my_user_name\docker-gl-runner> docker login gitlab.my_domain.tld:5050 -u my_user_name 
Password:
Error response from daemon: Get https://gitlab.my_domain.tld:5050/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

I do have a user account setup, I did generate a personal access token, and have tried both (username + password, username + token).

The service is listening on 5050:

[root@gitlab gitlab]# netstat -an | grep 50..
tcp        0      0 0.0.0.0:5050            0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:5000          0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:5001          0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:5001          127.0.0.1:46666         ESTABLISHED
tcp        0      0 127.0.0.1:5001          127.0.0.1:46560         TIME_WAIT
tcp        0      0 127.0.0.1:46666         127.0.0.1:5001          ESTABLISHED

And here are the relevant settings (I believe) in the gitlab.rb file:

[root@gitlab gitlab]# cat gitlab.rb | grep registry
# gitlab_rails['gitlab_default_projects_features_container_registry'] = true

registry_external_url 'https://gitlab.my_domain.tld'
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "gitlab.my_domain.tld"
gitlab_rails['registry_port'] = "5005"
gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry"
# gitlab_rails['registry_notification_secret'] = nil
# gitlab_rails['registry_api_url'] = "http://localhost:5000"
# gitlab_rails['registry_key_path'] = "/var/opt/gitlab/gitlab-rails/certificate.key"
# gitlab_rails['registry_issuer'] = "omnibus-gitlab-issuer"

registry['enable'] = true
# registry['username'] = "registry"
# registry['group'] = "registry"
# registry['uid'] = nil
# registry['gid'] = nil
registry['dir'] = "/var/opt/gitlab/registry"
registry['registry_http_addr'] = "localhost:5000"
registry['debug_addr'] = "localhost:5001"
registry['log_directory'] = "/var/log/gitlab/registry"
registry['env_directory'] = "/opt/gitlab/etc/registry/env"

registry['log_level'] = "info"
registry['log_formatter'] = "text"
# registry['rootcertbundle'] = "/var/opt/gitlab/registry/certificate.crt"
# registry['health_storagedriver_enabled'] = true
# registry['storage_delete_enabled'] = true
# registry['validation_enabled'] = false
# registry['autoredirect'] = false
# registry['compatibility_schema1_enabled'] = false

Any thoughts on what's different, messed up, or should be looked into? Or maybe even a, "Here's what mine looks like, compare it to yours" (I'll show you mine if you show me yours?)

Answer 1

Okay, so I figured out the problems.

First of all, Centos8, firewalld, blocking the connection. Had to allow the docker0 zone to be trusted. Forgot that one.

Second, tcpdump showed that the connection was not being ACK'd. Turns out, the docker service wasn't running. You can config gitlab all you want, but you need Docker to be running. Who knew?

Third, after fixing both of those problems, I was hit with the self-signed certificate problem. If you're curious for the simple way to solve it:

openssl s_client -showcerts -connect gitlab.my_domain.tld:443 </dev/null 2>/dev/null|openssl x509 -outform PEM >/etc/docker/certs.d/gitlab.my_domain.tld:5050/ca.crt

That'll pull down the cert from the remote host and trust it for Docker.