Fix: Shutting down the Docker Container, inspecting the reconfiguration logs (located in the logs/reconfiguration directory inside the installation folder) and making adjustments to the configuration file based on the errors apparent in the log.
Fixing the boot-loop (step by step)
NOTE: If you are not running as root, I recommend temporarily running as root (using sudo su) to avoid Permission denied errors.
After manually stopping the Docker Container I was able to view the full log located at /srv/gitlab/logs/reconfigure (with the latest timestamp)
Excerpt from the latest reconfiguration log:
[2018-05-19T00:24:50+00:00] INFO: *** Chef 13.6.4 ***
[2018-05-19T00:24:50+00:00] INFO: Platform: x86_64-linux
[2018-05-19T00:24:50+00:00] INFO: Chef-client pid: 25
[2018-05-19T00:24:50+00:00] INFO: The plugin path /etc/chef/ohai/plugins does not exist. Skipping...
[2018-05-19T00:24:52+00:00] WARN: Plugin Network: unable to detect ipaddress
[2018-05-19T00:24:52+00:00] INFO: Setting the run_list to ["recipe[gitlab]"] from CLI options
[2018-05-19T00:24:52+00:00] INFO: Run List is [recipe[gitlab]]
[2018-05-19T00:24:52+00:00] INFO: Run List expands to [gitlab]
[2018-05-19T00:24:52+00:00] INFO: Starting Chef Run for [REDACTED]
[2018-05-19T00:24:52+00:00] INFO: Running start handlers
[2018-05-19T00:24:52+00:00] INFO: Start handlers complete.
[2018-05-19T00:24:54+00:00] INFO: Loading cookbooks [gitlab@0.0.1, package@0.1.0, postgresql@0.1.0, registry@0.1.0, mattermost@0.1.0, consul@0.0.0, gitaly@0.1.0, letsencrypt@0.1.0, nginx@0.1.0, runit@0.14.2, acme@3.1.0, crond@0.1.0, compat_resource@12.19.0]
[2018-05-19T00:24:56+00:00] WARN: Runtime directory '/run' is not a tmpfs.
Error messages are a good start, but won't reveal the underlying issue...
[2018-05-19T00:24:56+00:00] ERROR: Running exception handlers
[2018-05-19T00:24:56+00:00] ERROR: Exception handlers complete
Now the last 3 lines of the log revealed the problem
[2018-05-19T00:24:56+00:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/chef-stacktrace.out
[2018-05-19T00:24:56+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2018-05-19T00:24:56+00:00] FATAL: RuntimeError: Unsupported GitLab Registry external URL path: /gitlab/registry
Now after seeing the very last line, I see that my configuration is invalid and is the cause of the fatal issue.
To fix the issue, I edited the configuration while the docker container was in an unbooted state (configuration file located at /srv/gitlab/config/gitlab.rb). After fixing the configuration, which in my case was to comment out all GitLab Registry configuration options (since I decided to wait with testing this feature for now).
Diff for configuration
################################################################################
## Container Registry settings
##! Docs: https://docs.gitlab.com/ce/administration/container_registry.html
################################################################################
- registry_external_url 'http://[REDACTED]:4567/gitlab/registry'
+ # registry_external_url 'http://[REDACTED]:4567/gitlab/registry'
### Settings used by GitLab application
- gitlab_rails['registry_enabled'] = true
+ # gitlab_rails['registry_enabled'] = true
- gitlab_rails['registry_host'] = "[REDACTED]"
+ # gitlab_rails['registry_host'] = "[REDACTED]"
- gitlab_rails['registry_port'] = "4567"
+ # gitlab_rails['registry_port'] = "4567"
- gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry"
+ # gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry"
I ran the following commands to boot the container and apply the new configuration.
docker start gitlab
sudo docker exec gitlab gitlab-ctl reconfigure
Promising results right away as the Container did not reboot right after I issued the command.
Starting Chef Client, version 13.6.4
resolving cookbooks for run list: ["gitlab"]
Synchronizing Cookbooks:
- gitlab (0.0.1)
- package (0.1.0)
- registry (0.1.0)
- postgresql (0.1.0)
- letsencrypt (0.1.0)
- mattermost (0.1.0)
- runit (0.14.2)
- nginx (0.1.0)
- gitaly (0.1.0)
- consul (0.0.0)
- acme (3.1.0)
- crond (0.1.0)
- compat_resource (12.19.0)
Installing Cookbook Gems:
...
That's it! Now everything works as it should. Turned out user error was the cause of the issue.
