Setup: GitLab installed with Omnibus package on CentOS 6.4
After I've made a change in the GitLab configuration file /etc/gitlab/gitlab.rb to configure Bitbucket as OAuth2 provider, I've run the reconfiguration:
sudo gitlab-ctl reconfigure
After I've launched this command I've seen some error, maybe about a monitoring service, but the SSH session drop and I haven't seen what happened. After restoring the session GitLab was not accessible, I was able to see the GitLab logo with a 502 error: "GitLab is taking too much time to respond" or something similar.
So I've restored the configuration file, thinking something was wrong in it...
Running tail I've got the below errors (Prometheus is failing):
2018-01-30_22:19:19.28021 time="2018-01-30T23:19:19+01:00" level=info msg="Starting prometheus (version=, branch=, revision=)" source="main.go:87"
2018-01-30_22:19:19.28027 time="2018-01-30T23:19:19+01:00" level=info msg="Build context (go=go1.9.2, user=, date=)" source="main.go:88"
2018-01-30_22:19:19.28030 time="2018-01-30T23:19:19+01:00" level=info msg="Host details (Linux 2.6.32-642.11.1.el6.x86_64 #1 SMP Fri Nov 18 19:25:05 UTC 2016 x86_64 lnx01.brainsen.com (none))" source="main.go:89"
2018-01-30_22:19:19.29486 time="2018-01-30T23:19:19+01:00" level=info msg="Loading configuration file /var/opt/gitlab/prometheus/prometheus.yml" source="main.go:254"
2018-01-30_22:19:19.29779 time="2018-01-30T23:19:19+01:00" level=error msg="Could not open the fingerprint-to-metric index for archived series. Please try a 3rd party tool to repair LevelDB in directory "/var/opt/gitlab/prometheus/data/archived_fingerprint_to_metric". If unsuccessful or undesired, delete the whole directory and restart Prometheus for crash recovery. You will lose all archived time series." source="persistence.go:213"
2018-01-30_22:19:19.29781 time="2018-01-30T23:19:19+01:00" level=error msg="Error opening memory series storage: leveldb: manifest corrupted (field 'comparer'): missing [file=MANIFEST-000923]" source="main.go:192"
I've run the command
sudo -u gitlab-prometheus python -c "import leveldb; leveldb.RepairDB('/var/opt/gitlab/prometheus/data/archived_fingerprint_to_metric')"
but I get the message:
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named leveldb
Now GitLab is working fine but running tail I still see the above error about Prometheus every 2 seconds and I'd like to fix it without breaking my GitLab installation.
