-2

Our Self-hosted GitLab server is randomly getting broken and we couldn't figure out why. This random behavior affects our deployments, it gets too slow. After restarting it, stays up for a few hours and goes down, throws 500 or 502 error. After bringing it back up, I see either the sidekiq or gitaly metrics on omnibus grafana dashboard goes down compared to other services. What do these services do and how to debug this issue? Sidekiq metric image Gitaly metric image

System Info:

  • OS - Centos 6.10 (final)
  • CPU - 8 cores
  • Memory usage - 8.82 GB / 15.6 GB
  • Disk Usage -
    • root : 111 GB / 389 GB
    • boot : 169 MB / 476 MB
    • /mnt/resource : 8.25 GB / 62.9 GB
Vikram Giriraj
  • 3
  • 2
  • [Accessibility](https://en.wikipedia.org/wiki/Computer_accessibility) has to do with how your application caters to people who use assistive technologies or may require other kinds of aids. It has nothing to do with your problem. – max Mar 14 '22 at 17:04
  • Ah come on, where's the benevolence here? He obviously means he cannot access it, are we here to help each others or to shoot down people at will? – tsadiq Mar 15 '22 at 09:53

1 Answers1

0

I've met the same kind of issue after a few months not using an instance i had setup for one of my customers. You might want to check that your instance is up to date and update it if not - some vulnerabilities might be at fault here.

In my case it was a weird process sucking all my cpu, i suspect some kind of crypto currency miner was ran using a known exploit that was fixed in a later update, all went better after i killed it and updated the version.

tsadiq
  • 402
  • 3
  • 8
  • yes. that's the issue. In logs I could see gitlab trying to kill few processes, but they come back as quickly as they get killed. crontab shows some wierd call to run a python script from pastebin.com executed by user 'git'. I've requested them to upgrade the server ASAP. Curious to know what steps you took before and after update? – Vikram Giriraj Mar 24 '22 at 05:55
  • Honestly best you can do if setup another server with latest version and migrate your projects to it then get rid of the old one ; i tried to upgrade mine (sameersbn docker) and got stuck to a point where postgres version changed and messed up my db, so it might be easier to just export / import your projects – tsadiq Mar 25 '22 at 14:16