3

We have a set of CI pipelines running in Gitlab v15.3.

We have an intermittent issue, where some build targets hang. If we cancel and retry they run to completion. There seems to be no distinct pattern; any targets might be affected, a set of parallel tasks will generally work but one or two jobs hang.

The symptom we see is

Checking out e18f49c1 as XYZ ...

Skipping Git submodules setup

and then no further action, hangs indefinitely. Cancel and retry always seems to work.

We have a relatively large mono-repository and a large gitlab-ci.yml file.

Suggestions please

djna
  • 54,992
  • 14
  • 74
  • 117
  • 1
    please share your pipeline to narrow down the options , this question is to broad IMO – lcarvajal Feb 14 '23 at 19:46
  • @lcarvajal we have compile, test and package stages, the hangs occur at the git checkout step before the stage-specific processing starts in any of those stages. I can't share our proprietary code. – djna Feb 15 '23 at 09:17
  • Not a gitlab expert but I use several similar tools and what you are describing could be an issue with a faulty agent (since it runs sometimes and works fine after reset), problematic artifact or project, inefficient build plan or even a recurring network issue. Your best bet is going to be to find the lowest level logfiles you can and see if you can trap the error so you can get an idea of where in the process it's dying. –  Feb 15 '23 at 16:27

1 Answers1

0

We have a relatively large mono-repository

Then consider updating your gitlab-ci.yml, optimizing GitLab for large repositories.

At least check if a shallow clone would help:

variables:
  GIT_DEPTH: 3

test:
  script:
    - ls -al

If not, try and split your gitlab-ci jobs into multiple files, to test each of them in isolation and to check if one of them would triggers the issue.

VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
  • Worth a try, but I'm not hopeful. Much of this seems to be applicable if things were unacceptably slow rather than intermittently hanging on the same task. As far as I can tell it will never progress. Then resubmit the job and proceeds seemlessly, then future runs may work or hang apparently at random. – djna Feb 10 '23 at 17:32
  • @djna I agree, this is a long shot, but I would still test if one of those two approaches can help. – VonC Feb 10 '23 at 18:25