18

I use Gitlab for doing Continuous Integration and Development and all of a sudden I get this error message "There has been a runner system failure, please try again"

enter image description here

There's no real error message or error code.

I've tried restarting the gitlab runner, using gitlab-runner restart, I've done a reboot of the server its running on but I keep getting this error message on Gitlab whenever I push a code change.

CAMD_3441
  • 2,514
  • 2
  • 23
  • 38

1 Answers1

24

After a couple of hours, I realized the issue is that the server that Gitlab Runner is running on has no space left.

I logged into the server in question. Looked at the Gitlab log file using the following command: journalctl -u gitlab-runner

And it showed me the following logs:

May 21 08:20:41 gitlab-runner[18936]: Checking for jobs... received                       job=178911 repo_url=https://.......git runner=f540b942
May 21 08:20:41 gitlab-runner-01 gitlab-runner[18936]: WARNING: Failed to process runner                   builds=0 error=open /tmp/trace543210445: no space left on device executor=docker runner=f540b942

To fix this issue I ran docker conatiner prune which clears out stopped containers.

Alternatively you could use docker system prune which would remove all unused objects.

See https://linuxize.com/post/how-to-remove-docker-images-containers-volumes-and-networks/ for more information about those docker commands.

Afterwards, I no longer got the error on Gitlab when pushing changes.

Naveed Kamran
  • 453
  • 1
  • 4
  • 16
CAMD_3441
  • 2,514
  • 2
  • 23
  • 38
  • 1
    Just to add a small information that may be valuable for others because it took me some research and time, i had the same problem, it was also related to disk space being full, in my case the mysql volume was huge compared to the data it holds. I had to purge the binlog older than 3 days in order for it to work. Hope this helps someone facing the same issue ! – tchartron Mar 22 '22 at 09:48