Gitlab CE Omnibus Started to Get 503 Error

Question

I've installed GitLab on a CentOS 7 using the Omnibus installer a few months ago. It has ran fine but, today it started to throw a 503 error page when I visit the main page to login.

I ran the gitlab-ctl status command and I don't see anything jumping out at me that shows there is a problem. Does anyone know where to go from here?

# gitlab-ctl status
run: alertmanager: (pid 4591) 14837s; run: log: (pid 4590) 14837s
run: gitaly: (pid 4580) 14837s; run: log: (pid 4579) 14837s
run: gitlab-monitor: (pid 4574) 14837s; run: log: (pid 4572) 14837s
run: gitlab-workhorse: (pid 4583) 14837s; run: log: (pid 4581) 14837s
run: logrotate: (pid 15028) 436s; run: log: (pid 4567) 14837s
run: nginx: (pid 4573) 14837s; run: log: (pid 4571) 14837s
run: node-exporter: (pid 4584) 14837s; run: log: (pid 4582) 14837s
run: postgres-exporter: (pid 4568) 14837s; run: log: (pid 4566) 14837s
run: postgresql: (pid 4593) 14837s; run: log: (pid 4592) 14837s
run: prometheus: (pid 4586) 14837s; run: log: (pid 4585) 14837s
run: redis: (pid 4562) 14837s; run: log: (pid 4560) 14837s
run: redis-exporter: (pid 4576) 14837s; run: log: (pid 4575) 14837s
run: sidekiq: (pid 4559) 14837s; run: log: (pid 4558) 14837s
run: unicorn: (pid 23104) 1s; run: log: (pid 4577) 14837s

I went on to use gitlab-ctl tail per suggestions in comments and found references to a stale unicorn pid file. Simply restarting all services with gitlab-ctl restart made the 503 error go away and all is well after that. Still not sure why unicorn decided to crash though.

Is the disk full? Have you looked at the logs? `/var/log/gitlab` or run `gitlab-ctl tail`. — Aaron Copley, Apr 08 '19 at 15:46
thanks, I did eventually find the tail command then looked in /var/log/gitlab/unicorn and found a stale unicorn pid file message — SciGuyMcQ, Apr 08 '19 at 16:40
It's ok to post a self-answer - it will be very usable for future readers. — kubanczyk, Apr 09 '19 at 07:54

score 1 · Answer 1 · answered Apr 12 '19 at 13:12

I was able to fix the issue to get back up and running but, I'm unsure of the root cause.

first I used gitlab-ctl tail and found that there was a log message stating there was a stale unicorn PID file. This basically told me that unicorn application server had crashed for some reason or another.

I restarted all gitlab services with gitlab-ctl restart which takes a few minutes to properly shutdown and restart all services and eventually I was able to use GitLab under normal conditions again.

Gitlab CE Omnibus Started to Get 503 Error

1 Answers1