Gitlab CI: run job only if artifact exists

Question

I have monorepo and I want to run child pipeline depending on content of directory which has changed. In job prepare_config I check where are latest changes, I create child config yml and in next stage's job run_child I run child pipeline from .

The problem is, if model-gitlab-ci.yml doesn't exist, then job run_child fails instead of skipping due to missing artifact. I searched for solution to conditionally run job only if artifact exists instead of failing, but didn't found any solution. Maybe someone here have some idea?

.gitlab-ci.yml:

stages:
  - .pre
  - build

prepare_config:
  stage: .pre
  tags:
    - sometag
  rules:
    - if: $CI_COMMIT_TAG == null
      when: always
      changes:
      - '.gitlab-ci.yml'
      - 'DIR_A/**/*'
      - 'DIR_B/**/*'
      - 'DIR_C/**/*'
  script:
    - |-
      files=$(git diff-tree --name-only --no-commit-id ${CI_MERGE_REQUEST_TARGET_BRANCH_SHA:-$CI_COMMIT_SHA})
      echo "Files changed: $files"
      for f in $files; do
        if [ -d $f ]; then
          sed "s/{{ MODEL_NAME }}/$f/g" .gitlab-ci-template.yml >> model-gitlab-ci.yml
        fi
      done
  artifacts:
    paths:
      - "model-gitlab-ci.yml"

run_child:
  stage: build
  rules:
    - if: $CI_COMMIT_TAG == null
      when: always
  needs:
    - job: prepare_config
      artifacts: true
  trigger:
    include:
      - artifact: model-gitlab-ci.yml
        job: prepare_config
    strategy: depend

Can't you set variable in `prepare_config` say variable as `IS_MODEL_CI_EXISTS` with 0 (i.e. no) or 1 (i.e. yes) value. Furthermore, in `run_child` check value of `IS_MODEL_CI_EXISTS` and based on that proceed with execution or skip execution of `run_child` ? — amitd, Feb 02 '21 at 22:34
@amitd how do you want to pass that variable between stages? If you meant `rules:if` then no, it won't work because it is being evaluated once pipelineis created. — Taz, Feb 02 '21 at 22:39
Dynamically created jobs could be a solution (https://docs.gitlab.com/ee/ci/parent_child_pipelines.html#dynamic-child-pipelines). For creating dynamic child pipelines you could use jsonnet (https://jsonnet.org/). — Jakob Liskow, Feb 04 '21 at 21:47
@JkbLskw did you take a look at my yml? IT IS dynamic pipeline. — Taz, Feb 05 '21 at 13:00
@Taz sure. With this kind of dynamic pipeline you cannot react to artifacts/variables created in script-sections of other jobs. With jsonnet, jobs can be created using variables in script-sections. This would help you with your problem. — Jakob Liskow, Feb 05 '21 at 18:37
@JakobLiskow How does jsonnet help with dynamic pipelines? jsonnet is strictly a template language AFAICT. — erstaples, May 12 '21 at 15:56
@JakobLiskow you still don't get it. The case is simple: if artifact exists, then downstream job runs, if not it does not. Neither `jsonnet`, nor any other scripting language can be solution here. — Taz, Jun 30 '21 at 21:02
@Taz with a language like jsonnet, however, a job can be created or not based on a value. this behavior can then be used to trigger or not to trigger your generated job in your generated yaml file. Its that simple. — Jakob Liskow, Jul 01 '21 at 21:40
@Jakob Liskow if you say so, please provide answer to my question with solution you're describing. — Taz, Jul 01 '21 at 22:15

Jakob Liskow · Answer 1 · 2021-07-15T06:26:08.683

The job "jsonnet" uses jsonnet to generate a YAML file "trigger.yml" from the file "trigger.jsonnet". If the variable in the JSONNET file triggering == true it contains the jobs "artifact:helper" and "trigger". If triggering == false it contains the job "not:triggered", which runs empty.

The job "artifact:helper" is required so that the generated file "generated.yml" can be transferred to the trigger as an artifact. If a trigger job in a child pipeline wanted to use an artifact of the parent pipeline directly, a cross-dependency exception would occur.

The job "trigger" in the YAML file "trigger.yml" ultimately triggers the job "template_job" from "generated.yml".

Jobs if triggering == true

Jobs if triggering == false

.gitlab-ci.yml

jsonnet:
  image: jkblskw/jsonnet:jsonnet-curl
  stage: .pre
  script:
    # do something
    - if [ "$TEST_TRIGGER" = true ]; then TRIGGERING=true && cp template.yml generated.yml; else TRIGGERING=false; fi
    - jsonnet --ext-code triggering=$TRIGGERING --ext-code pipeline_id=$CI_PIPELINE_ID trigger.jsonnet > trigger.yml
  artifacts:
    paths:
      - trigger.yml
      - generated.yml

trigger:
  stage: build 
  trigger:
    include:
      - artifact: trigger.yml
        job: jsonnet
    strategy: depend

trigger.jsonnet

local triggering = std.extVar("triggering");
local pipeline_id = std.extVar("pipeline_id");

if triggering then 
{
    ["artifact:helper"]: {
        stage: "build",
        variables: {
            PARENT_PIPELINE_ID: pipeline_id
        },
        needs:{
            pipeline: "$PARENT_PIPELINE_ID",
            job: "jsonnet"
        },
        script: "echo 'Job needs to be executed in order for the artifact to be triggered. A trigger in the same job throws a cross-reference exception.'",
        artifacts: {
            paths: ["generated.yml"]
        }
    },
    ["trigger"]: {
        stage: "test",
        needs: ["artifact:helper"],
        trigger: {
            include: {
                artifact: "generated.yml",
                job: "artifact:helper"
            },
            strategy: "depend"
        }
    } 
}else
{
    ["not:triggered"]: {
        stage: "build",
        script: "echo 'no further triggering'"
    } 
}

template.yml

template_job:
  stage: test
  script: 
      - echo "running template_job"

Gitlab CI: run job only if artifact exists

1 Answers1

Linked