Can I tell a Gitlab-CI-job to retrieve artifacts from an older pipeline?

Question

I have a gitlab-ci configuration that looks something like this:

simulate:
  stage: simulate
  when: manual
  script: 
    - "do simulation in matlab"
  artifacts:
    paths:
    - results/*.mat

 analyze:    
   stage: analyze
   script: 
     - "do analysis in matlab"
   dependencies:
     - simulate

The analyze job needs the artifact from the simulate job. However, the simulate job takes several hours to run, thus, I made it manual, so it does not run every time the pipeline runs. The situation as it is: If I do not run the simulation, I can not run the analysis. (The analysis makes no sense without the data from the simulation.)

Let's say I did not change the simulation code, just the analysis code. Then it would be convenient to run the analyze job with the artifact of a older pipeline. Is this possible? And how?

Does this answer your question? [Gitlab: How to use artifacts in subsequent jobs after build](https://stackoverflow.com/questions/56721584/gitlab-how-to-use-artifacts-in-subsequent-jobs-after-build) — Arty-chan, Jan 08 '22 at 17:48

score 0 · Answer 1 · edited Jan 15 '22 at 04:45

0

You can use the dependencies option to download artifacts or you can also use the needs keyword.

You can use needs something like below in your code:

simulate:
  stage: simulate
  when: manual
  script: 
    - "do simulation in matlab"
  artifacts:
    paths:
    - results/*.mat

 analyze:    
   stage: analyze
   needs: [simulate]
   script: 
     - "do analysis in matlab"

edited Jan 15 '22 at 04:45

Sty

760
1
9
30

answered Jan 09 '22 at 04:22

John

22
3

`needs` and `dependencies` keywords do not solve my problem. When I don't run the simulate job in the current pipeline, the analyze job still has no data to execute on. – tiga21 Jan 11 '22 at 13:50

Sty · Accepted Answer · 2022-01-24T15:24:24.613

If what you're looking for is to be able to use results of earlier pipelines in the present one, then you may want to look into the cache directive of the CI definitions, as they carry over more intuitively between pipelines than artifacts.

Like so:

simulate:
  stage: simulate
  when: manual
  script: 
    - "do simulation in matlab"
  cache:
    key: simulation-results
    policy: push
    paths:
    - results/*.mat

 analyze:    
   stage: analyze
   script: 
     - "do analysis in matlab"
   cache:
     key: simulation-results
     policy: pull
     paths:
     - results/*.mat

If you have a premium Gitlab licence, you could apparently also directly access the artifact using the CI_JOB_TOKEN, but that may be of limited use in your scenario if your pipeline runs occasionally, since then you'd likely have to adjust the CI definitions accordingly after every new source data pipeline.

Thank you. This is exactly what I was looking for! – tiga21 Jan 19 '22 at 14:15 — tiga21, Jan 19 '22 at 14:15

Can I tell a Gitlab-CI-job to retrieve artifacts from an older pipeline?

2 Answers2