Separate Jobs in CI Pipeline result in very long pipeline run

Question

I had a version of the following pipeline that ran as a single job, and all in this took under 6 minutes. I then updated the pipeline to be broken up into separate Jobs to make it easier to know which has failed, and now the total pipeline is 15+ minutes. I can only assume each Job is considered its own pipeline, with tear up/down process which is taking a long time across four different jobs. I'm looking for advice on how to refactor this GitLab CI to get back to my original 6 or less length of time:

image: cirrusci/flutter:stable

before_script:
  - flutter pub get
  - flutter clean
  - flutter --version

stages:
  - build-aot
  - analyze
  - format-check
  - test-on-machine-with-coverage

build-aot:
  stage: build-aot
  script:
    - flutter build aot
  only:
    - master
    - merge_requests
  except:
     variables:
       - $CI_MERGE_REQUEST_TARGET_BRANCH_NAME != "master" 

analyze:
  stage: analyze
  script:
    - flutter analyze
  only:
    - master
    - merge_requests
  except:
     variables:
       - $CI_MERGE_REQUEST_TARGET_BRANCH_NAME != "master" 

format-check:
  stage: format-check
  script:
    - flutter format --set-exit-if-changed lib test
  only:
    - master
    - merge_requests
  except:
     variables:
       - $CI_MERGE_REQUEST_TARGET_BRANCH_NAME != "master" 

test-on-machine-with-coverage:
  stage: test-on-machine-with-coverage
  script:
    - flutter pub global activate junitreport
    - export PATH="$PATH":"$HOME/.pub-cache/bin"
    - flutter test --machine | tojunit -o report.xml
    - flutter test --coverage ./lib 
    - lcov -r coverage/lcov.info '*/__test*__/*' -o coverage/lcov_cleaned.info
    - genhtml coverage/lcov_cleaned.info --output=coverage
  artifacts:
    when: always
    paths:
      - rspec.xml
      - coverage
    reports:
      junit:
        - report.xml
  only:
    - master
    - merge_requests
  except:
     variables:
       - $CI_MERGE_REQUEST_TARGET_BRANCH_NAME != "master"

score 0 · Accepted Answer · answered Feb 18 '21 at 06:05

A pipeline is commonly defined as a series of jobs (According to Gitlab's documentation), so in order to properly script your repository's CI/CD where splitting your jobs functions keeps a similar performance to when they were together, a common practice is to implement caching between stages.

Caching allows you to properly isolate each job runtime environment while simultaneously optimizing speed and bandwidth as it avoids downloading or building the same files over multiple times.

Now let's take a look at your gitlab-ci.yml code, as to implement caching for optimizing your pipeline execution time. Flutter's documentation says a global cache is created once you install each package, and therefore you only have to install once per build, and it is located at .pub-cache:

image: cirrusci/flutter:stable

variables:
  PUB_CACHE: $CI_PROJECT_DIR/.pub-cache #https://docs.gitlab.com/ee/ci/yaml/README.html#cachepaths

cache:
  key: ${CI_COMMIT_REF_SLUG} #Only use one cache per whole pipeline
  paths:
    - .pub-cache/

before_script:
  - flutter pub cache --all #https://dart.dev/tools/pub/cmd/pub-cache
  - flutter --version

stages:
  - build-aot
  - analyze
  - format-check
  - test-on-machine-with-coverage

build-aot:
  stage: build-aot
  script:
    - flutter build aot
  only:
    - master
    - merge_requests
  except:
    variables:
      - $CI_MERGE_REQUEST_TARGET_BRANCH_NAME != "master" 
    
analyze:
  stage: analyze
  script:
    - flutter analyze
  only:
    - master
    - merge_requests
  except:
     variables:
       - $CI_MERGE_REQUEST_TARGET_BRANCH_NAME != "master" 

format-check:
  stage: format-check
  script:
    - flutter format --set-exit-if-changed lib test
  only:
    - master
    - merge_requests
  except:
     variables:
       - $CI_MERGE_REQUEST_TARGET_BRANCH_NAME != "master" 

test-on-machine-with-coverage:
  stage: test-on-machine-with-coverage
  script:
    - flutter pub global activate junitreport
    - export PATH="$PATH":"$HOME/.pub-cache/bin"
    - flutter test --machine | tojunit -o report.xml
    - flutter test --coverage ./lib 
    - lcov -r coverage/lcov.info '*/__test*__/*' -o coverage/lcov_cleaned.info
    - genhtml coverage/lcov_cleaned.info --output=coverage
  artifacts:
    when: always
    paths:
      - rspec.xml
      - coverage
    reports:
      junit:
        - report.xml
  only:
    - master
    - merge_requests
  except:
     variables:
       - $CI_MERGE_REQUEST_TARGET_BRANCH_NAME != "master"

This will still download your packages every new series of pipelines (when committing a change), removing the key: flag should keep the cache until you manually erase it on Gitlab's interface.

Separate Jobs in CI Pipeline result in very long pipeline run

1 Answers1