Gitlab CI/CD fails while "Cleaning up project directory and file based variables" with "ERROR: Job failed: exit code 1"

Question

My gitlab pipeline which has been running for nearly six months is now failing unexpectedly.

Every line prior runs successfully and then this happens:

Setting up curl (7.52.1-5+deb9u16) ...
$ curl -s https://deb.nodesource.com/setup_12.x | bash
Cleaning up project directory and file based variables 
ERROR: Job failed: exit code 1

I cannot for the life of me figure out what has changed. I thought it might be related this issue but I'm not having any network issues, timeouts, etc.

Mildly obfuscated version of my .gitlab-ci.yml. Obviously I'm using .gitlab-ci.yml to configure my pipelines and I'm also using the shared GitLab runners.


image: python:3.6-stretch

variables:
    ACCESS_KEY_ID: **********
    SECRET_ACCESS_KEY: **********

before_script:
  - apt-get update
  - apt-get install -y curl
  - curl -s https://deb.nodesource.com/setup_12.x | bash
  - apt-get install -y nodejs
  - apt-get install -y npm
  - npm install -g serverless
  - pip install  --upgrade awscli
  - python --version
  - nodejs --version

stages:
  - deploy

deploy:
  stage: deploy

  only:
  - master   # We will run the CD only when something is going to change in master branch.

  script:
    - npm install   # Archive the code repository.
    - pip install -r requirements.txt

    - cd services/service1/
    - sls deploy -v --stage production
    - cd ../../

    - cd services/service2/
    - sls deploy -v --stage production
    - cd ../../

    - cd services/service3/
    - sls deploy -v --stage production
    - cd ../../


  environment:
    name: master

If you're using the shared GitLab Runners provided when using gitlab.com (opposed to your own, self-hosted GitLab instance) then you should contact support / [raise an Issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new). This error doesn't seem to be anything with your pipeline definition. — Adam Marshall, Oct 01 '21 at 20:13

score 4 · Answer 1 · answered Oct 02 '21 at 02:04

That second to last line (Cleaning up project directory and file based variables ) is always present in a CI/CD job, pass or fail.

What's likely happening is the last command, curl -s https://deb.nodesource.com/setup_12.x | bash is failing. Unfortunately, since you're downloading a remove file and piping it into bash, it's quite possible that your pipeline starts randomly failing, because that script isn't guaranteed to be the same every time.

To test it out, I created a clean ubuntu VM, and ran that curl command, and get the following error:

Your best bet to fix this long-term is to create a container that has all the dependencies you need for you CI baked in, and store that in your container registry for your GitLab project, then to pull that container each time. Not only will that save you CI/CD minutes since you don't have to run installs each time, but it'll prevent this exact issue where your dependencies change underneath you and cause error. It's also worth noting you should be very careful about passing an externally downloaded script to bash, because that script could change to include anything and your CI would just unknowingly run it.

Adiii · Answer 2 · 2022-02-20T06:35:28.167

I would like to share my case so might help someone else. Base on my experience this error is mostly related to the docker image, as after this stage the pipeline kicks the docker image to start the container.

Was getting the same error

Cleaning up project directory and file based variables

In my case, I build the desired image build on Mac m1 while the runner was linux.

package:
  stage: package
  image:
    name: kaniko:curl
    entrypoint: [""]

So you can run the docker image and if that working on the underlying OS of the runner then it should work.

But in your case I would recommend to move the before stage to pre-build image, as it creates overhead on every pipeline, I do not see issue while running the command with the given docker image

before_script:
  - apt-get update
  - apt-get install -y curl
  - curl -s https://deb.nodesource.com/setup_12.x | bash
  - apt-get install -y nodejs
  - apt-get install -y npm
  - npm install -g serverless
  - pip install  --upgrade awscli

move them pre-build image.

btw Its working for me

stages:
  - deploy

before_script:
  - apt-get update
  - apt-get install -y curl
  - curl -s https://deb.nodesource.com/setup_12.x | bash
  - apt-get install -y nodejs
  - apt-get install -y npm
  - npm install -g serverless
  - pip install  --upgrade awscli
  - python --version
  - nodejs --version



deploy:
  stage: deploy
  tags:
    - kubernetes
  image: python:3.6-stretch
  script: 
    - echo "working"

Gitlab CI/CD fails while "Cleaning up project directory and file based variables" with "ERROR: Job failed: exit code 1"

2 Answers2