GitLab projects failed their last repository check

Question

I have received a gitlab alert mail stating that "One project failed its last repository check". I did check the error in Gitlab admin panel as "Last repository check (just now) failed. See the 'repocheck.log' file for error messages." As suggested in Admin Panel in Gitlab, i have checked the repocheck.log file and the error is as below.

"Could not fsck repository: error: Could not read 0f188244898707e6090498bc03aafd8ac25e776e failed to parse commit 0f188244898707e6090498bc03aafd8ac25e776e from object database for commit-graph error: Could not read 4ab7111f3f8f1083cee8e33ec033c18edfefb0e9"

This happened the same with another repo last week. Even that had similar error message and it is not resolved yet. Tried to clone the same repo in another gitlab instance to recreate the issue but the repo check there seems to be fine. Unable to find proper solution for that. Could any one please help on this.

Answer 1

It seems to be an active issue: https://gitlab.com/gitlab-org/gitaly/-/issues/2359#note_966195929 The issue is a couple of years old, but the note I link to, has updates from today, including info on fixes and workarounds.

In any case: The commit-graph is a relatively new feature and since it is a form of index into packed data, it can be recreated. So the corrupted repo can be easily fixed without loss of data.

Answer 2

I've now come across and fixed this twice, I found the fix on the forum:

1. Find all your failed repos at your instance's admin panel: https://your-gitlab-instance/admin/projects?last_repository_check_failed=1
2. Open the repo from that page and copy down its Gitaly relative path, it should look like @hashed/d4/73/c530f048efdf2711df6fa15198ff48003583303624f8b97c174fadc2cab5e582.git

NOTE! The following commands should be run with the user that gitlab runs as. You can do this with sudo su [username], in my case the username was git. They will work with sudo, but it may create files or folders that the gitlab user won't be able to access.

3. Run the fsck command using the repo's relative path (This should output the same text as your /var/log/gitlab/gitlab-rails/repocheck.log file):

/opt/gitlab/embedded/bin/git -C /var/opt/gitlab/git-data/repositories/[Gitlab relative path] fsck
# example
/opt/gitlab/embedded/bin/git -C /var/opt/gitlab/git-data/repositories/\@hashed/d4/73/c530f048efdf2711df6fa15198ff48003583303624f8b97c174fadc2cab5e582.git fsck

4. Run the gc command using the repo's relative path. This may take a minute to complete.

/opt/gitlab/embedded/bin/git -C /var/opt/gitlab/git-data/repositories/[Gitlab relative path] gc
# example
/opt/gitlab/embedded/bin/git -C /var/opt/gitlab/git-data/repositories/\@hashed/d4/73/c530f048efdf2711df6fa15198ff48003583303624f8b97c174fadc2cab5e582.git gc

5. Run the fsck command again from step 3 to check that the command was successful
6. On the repo's page from step 2, click the blue Trigger repository check button

Answer 3

It can be reproduced like this:

1. Visit a commit graph: Project sidebar / Repository / Graph
2. Delete one of the visible commits by rewriting history:

git checkout <branch>
git reset head~1 
git commit -am "Replace last commit with a new one" 
git push --force

3. Refresh the page of step 1

Now GitLab will send you "A commit graph at GitLab projects failed their last repository check" when it runs the repository check on a scheduled interval.

I'd consider it a bug that admins and maintainers get a warning about this, since nothing is wrong really. See the GitLab issues linked by Anders Bandholm for more details.