2

I am setting up GitLab Enterprise Edition for my organisation. We are using AWS EFS for storing repo data and two instances to serve SSH and HTTP/HTTPS traffic via AWS ELB. EFS volume is shared between both instances. But when I visit project home page after service restart we see 500 error page. We tried running gitlab-ctl reconfigure && gitlab-ctl restart but did not help.

  1. GitLab EE Version - 8.12.1-ee
  2. 2 * Centos 7 (AWS EC2 Instance - m4.xlarge)
  3. AWS ELB to load balance ssh and http/https traffic

Error in logs

     ==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/itops/docker" for 10.0.4.246 at 2016-09-26 07:58:58 +0000
Processing by ProjectsController#show as HTML
  Parameters: {"namespace_id"=>"itops", "id"=>"docker"}
Completed 500 Internal Server Error in 145ms (ActiveRecord: 14.2ms)

OpenSSL::Cipher::CipherError (bad decrypt):
  app/models/project.rb:531:in `import_url'
  app/models/project.rb:567:in `external_import?'
  app/models/project.rb:559:in `import?'
  app/models/project.rb:575:in `import_in_progress?'
  app/controllers/projects_controller.rb:95:in `show'
  lib/gitlab/request_profiler/middleware.rb:15:in `call'
  lib/gitlab/middleware/go.rb:16:in `call'
  lib/gitlab/middleware/readonly_geo.rb:29:in `call'


  ==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/itops/docker/refs/master/logs_tree/?offset=50&_=1474877373459" for 10.0.9.106 at 2016-09-26 08:09:40 +0000
Processing by Projects::RefsController#logs_tree as JS
  Parameters: {"offset"=>"50", "_"=>"1474877373459", "namespace_id"=>"itops", "project_id"=>"docker", "id"=>"master"}
Started GET "/itops/docker/commits/master" for 10.0.4.246 at 2016-09-26 08:09:40 +0000
Processing by Projects::CommitsController#show as HTML
  Parameters: {"namespace_id"=>"itops", "project_id"=>"docker", "id"=>"master"}
Completed 500 Internal Server Error in 176ms (ActiveRecord: 16.8ms)

ActionView::Template::Error (bad decrypt):
    1: - if @project.mirror?
    2:   .prepend-top-default.gray-content-block.second-block.clearfix
    3:     - import_url = @project.safe_import_url
    4:     .oneline.project-mirror-text
    5:       This project is mirrored from #{link_to import_url, import_url}.
    6:
  app/models/project.rb:531:in `import_url'
  app/models/project.rb:587:in `safe_import_url'
  app/views/projects/commits/_mirror_status.html.haml:3:in `_app_views_projects_commits__mirror_status_html_haml___2300853211630423479_63495260'
  app/views/projects/commits/show.html.haml:35:in `_app_views_projects_commits_show_html_haml___2695203885719227520_58528080'
  app/controllers/projects/commits_controller.rb:27:in `show'
  lib/gitlab/request_profiler/middleware.rb:15:in `call'
  lib/gitlab/middleware/go.rb:16:in `call'
  lib/gitlab/middleware/readonly_geo.rb:29:in `call'

I imported Docker project from GitHub. All other pages are working fine (tree/master , activity, Graph, Pipeline, issues) only project home page and docker/commits/master is showing error.

TylerH
  • 20,799
  • 66
  • 75
  • 101
Nischay
  • 105
  • 2
  • 11
  • Sounds like a general AWS health/network-communication issue. – Warren P Sep 26 '16 at 15:26
  • 500 erros is from GitLab Application. All instances are healthy and inservice. – Nischay Sep 26 '16 at 17:03
  • If this is an omnibus gitlab, use this diagnostic tool to output relevant problems: `sudo gitlab-rake gitlab:check SANITIZE=true` it might tell you what is wrong – Warren P Sep 26 '16 at 18:23

3 Answers3

2

Did you migrate from another server?

If so, it's possible your previous /etc/gitlab/gitlab-secrets.json file contains encryption keys that your new installation doesn't.

Copying over the old gitlab-secrets.json to /etc/gitlab on your new server (after making a backup of your new version, of course, in case you need to roll back) should fix it.

Lee Benson
  • 11,185
  • 6
  • 43
  • 57
0

I my case the same exception was rised when i create repository by importing from another. After shutting down the donor project server I catch 500 error. The exception must be import_url unavailable , not ssl error. My resolution was to clear broken link from database:

update projects set import_url=NULL where id = MY-PROJECT_ID;
shcherbak
  • 738
  • 8
  • 14
0

We contacted both AWS and gitlab support team for this issue and find out that AWS EFS(Elastic File System - fancy NFS service) is not compatible with Gitlab. For more info please visit below link.

https://docs.gitlab.com/ee/administration/high_availability/nfs.html#aws-elastic-file-system

  • The distributed architecture of EFS results in a higher latency per file operation

  • Gitlab process very small files in a serialised manner (Examples of serialized commands: ls, cp, tar, rsync, git)

  • EFS bases allowed IOPS on volume size. The larger the volume, the more IOPS are allocated. For smaller volumes, users may experience decent performance for a period of time due to 'Burst Credits'. Over a period of weeks to months credits may run out and performance will bottom out.

  • For larger volumes, allocated IOPS may not be the problem. Workloads where many small files are written in a serialized manner are not well-suited for EFS. EBS with an NFS server on top will perform much better.

Nischay
  • 105
  • 2
  • 11