dynamically setting artifact path/folder structure on gitlab-ci.yml

Question

I have the following gitlab-ci.yml file that reads the package.json using the jq processor to dynamically set the variable name of the artifact folder, something along the lines of

image: node:latest

stages:
 - build

before_script:
    ## steps ignored for purpose of question
    - export NAME_OF_ARTIFACT_FOLDER=$(cat package.json | jq -r .name)"_"$(cat package.json | jq -r .version)".zip"
    - echo $NAME_OF_ARTIFACT_FOLDER ##prints the expected name here eg. myApp_1.0.0.zip

prod_build:
  stage: build
  script:
   - echo $NAME_OF_ARTIFACT_FOLDER ##prints the expected name here eg. myApp_1.0.0.zip
   - yarn run build
  artifacts:
    paths:
    - dist/$NAME_OF_ARTIFACT_FOLDER ## this does not work
    expire_in: 2 hrs

The issue here is - dist/$NAME_OF_ARTIFACT_FOLDER does not work, not sure if am missing something here.

EDIT

Upon hard coding the expected path such as the following, it works fine, which would mean that the folder name is valid and that the artifact is indeed identified appropriately, but does NOT work when coming from $NAME_OF_ARTIFACT_FOLDER

 artifacts:
    paths:
    - dist/myApp_1.0.0.zip ##hardcoding the expected works just fine
    expire_in: 2 hrs

**for the same reasons as anyone not wanting to hard code the folder name generated dynamically and or subjected to change**. Not sure if you read the entire question. tl-dr- I capture the name and version of the project from package.json to generate the zipped artifact folder as part of `yarn build` and would want gitlab to identify the said artifact for deployment — Jaya, Apr 04 '18 at 21:19
@Jaya were you able to make this work somehow? Or do I really have to hardcode my .zip in build stage?? omg — trainoasis, Feb 19 '21 at 09:07
Unfortunately no, I did not. Atleast not at that time of posting the question (almost 3 yrs back). Hoping things have changed and gitlab has incorporated the feature in it. Good luck @trainoasis. — Jaya, Feb 19 '21 at 22:16

score 3 · Answer 1 · answered Apr 04 '18 at 20:27

3

Well, that is not possible currently. Manual says as follows:

The artifacts:name variable can make use of any of the predefined variables.

That is no variables set in your script part of the job can be used.

answered Apr 04 '18 at 20:27

Jakub Kania

15,665
2
37
47

3

My question is about the artifacts path and not the name. am confused – Jaya Apr 04 '18 at 20:31
@Jaya Oh, yeah you're right. Same thing goes, you can't use anything outside of predefined variables. – Jakub Kania Apr 05 '18 at 07:08

score 3 · Answer 2 · answered Mar 29 '23 at 10:08

While @Jakub Kania was not that wrong with it's summary

https://stackoverflow.com/a/49659852/529977 That is no variables set in your script part of the job can be used.

and I have to redict my own comment beneath @Matt Bracewell's post

https://stackoverflow.com/a/66707954/529977 I don't see this will help anything because it addresses artifacts:name but not artifacts:paths . paths is still not dynamically usable as you can see in https://docs.gitlab.com/ee/ci/variables/where_variables_can_be_used.html

~~the undocumented part says:__~~

you can use any variable in artifacts:path that is available at job level

i.e.

  prod_build:
+   variables:
+      NAME_OF_ARTIFACT: "myApp_*.zip"
    stage: build
    script:
     - echo $NAME_OF_ARTIFACT
     - yarn run build
    artifacts:
      paths:
      - dist/$NAME_OF_ARTIFACT
      expire_in: 2 hrs

That's indeed consequent to what is asked in Artifacts Filename Cannot Be Set with Dynamic Variables even they don't ask for paths but names

generate for myself:
  script:
    - mkdir build
    - |
      cat > build/artifact1$$.txt <<EOF
      Hello World! ${CI_PROJECT_ID} ${CI_PIPELINE_IID}
      EOF
    - export MY_VAR_FILE_NAME=artifact1$$
    - echo "MY_VAR_FILE_NAME=${MY_VAR_FILE_NAME}" >> generate.env
    - |
      cat > build/artifact2$$.txt <<EOF
      The end is near! ${CI_PROJECT_ID} ${CI_PIPELINE_IID}
      EOF
    - export MY_VAR_FILE2_NAME=artifact2$$
    - echo "MY_VAR_FILE2_NAME=${MY_VAR_FILE2_NAME}" >> generate.env
    - |
      cat > build/ci${CI_PIPELINE_IID}.txt <<EOF
      Happy to hear you! ${CI_PROJECT_ID} ${CI_PIPELINE_IID}
      EOF
  artifacts:
    reports:
      # using the https://docs.gitlab.com/ee/ci/yaml/artifacts_reports.html#artifactsreportsdotenv 
      # does NOT do the trick to fill a job variable before artifact:paths is evaluated!
      dotenv: generate.env
    paths: 
      - build/ci${CI_PIPELINE_IID}.txt # WORKS!
      - build/$MY_VAR_FILE_NAME.txt    # doen't work
      - build/$MY_VAR_FILE2_NAME.txt   # doen't work

⚠️ unknown variables are expanded to empty and build/$MY_VAR_FILE_NAME.txt becomes build/ i.e. all files in build folder are artifacts - mentioned in pipeline out WARNING: build/.txt: no matching files. Ensure that the artifact path is relative to the working directory resp. unexpected build/: found 4 matching files and directories

As a workaround - but yes, it's ugly -, you can do the trick either by using wildcards (but I assume you have good reasons to do not)

prod_build
  artifacts:
    paths:
      - dist/*.zip

OR splitting into two jobs

generate before save:
  script:
    - mkdir build
    - |
      cat > build/artifact1$$.txt <<EOF
      Hello World! ${CI_PROJECT_ID} ${CI_PIPELINE_IID}
      EOF
    - export MY_VAR_FILE_NAME=artifact1$$
    - echo "MY_VAR_FILE_NAME=${MY_VAR_FILE_NAME}" >> generate.env
    - |
      cat > build/artifact2$$.txt <<EOF
      The end is near! ${CI_PROJECT_ID} ${CI_PIPELINE_IID}
      EOF
    - export MY_VAR_FILE2_NAME=artifact2$$
    - echo "MY_VAR_FILE2_NAME=${MY_VAR_FILE2_NAME}" >> generate.env
  artifacts:
    reports:
      # using the https://docs.gitlab.com/ee/ci/yaml/artifacts_reports.html#artifactsreportsdotenv trick to fill a job variable for downstream of successor job
      dotenv: generate.env
    paths: 
      - build/*.txt

save artifacts:
  variables:
    MY_VAR_FILE_NAME: #filled by dotenv
    MY_VAR_FILE2_NAME: #filled by dotenv
  needs:
    - generate before save
  script:
    - find build/
    - echo "expecting … "
    - echo "- $MY_VAR_FILE_NAME"
    - echo "- $MY_VAR_FILE2_NAME"
  artifacts:
    paths: 
      - build/$MY_VAR_FILE_NAME.txt    # WORKS
      - build/$MY_VAR_FILE2_NAME.txt   # WORKS

i.e. in your case:

image: node:latest

stages:
 - build

pre_prod_build:
  stage: build
  script:
    ## steps ignored for purpose of question
    - export NAME_OF_ARTIFACT_FOLDER=$(cat package.json | jq -r .name)"_"$(cat package.json | jq -r .version)".zip"
    - echo $NAME_OF_ARTIFACT_FOLDER ##prints the expected name here eg. myApp_1.0.0.zip
    - echo "NAME_OF_ARTIFACT_FOLDER=${NAME_OF_ARTIFACT_FOLDER}" >> artifact_name.dotenv
  artifacts:
    reports:
      dotenv: artifact_name.dotenv

prod_build:
  stage: build
  variables:
    NAME_OF_ARTIFACT_FOLDER: # set by pre_prod_build dotenv
  needs:
    - job: pre_prod_build
  script:
   - echo $NAME_OF_ARTIFACT_FOLDER ##prints the expected name here eg. myApp_1.0.0.zip
   - yarn run build
  artifacts:
    paths:
    - dist/$NAME_OF_ARTIFACT_FOLDER ## NOW IT WORKS
    expire_in: 2 hrs

A great thanks @childno͡.de ! Your workaround is perfect ! – granier Aug 04 '23 at 13:16 — granier, Aug 04 '23 at 13:16
you're welcome @granier ;) – childno͡.de Sep 01 '23 at 13:01 — childno͡.de, Sep 01 '23 at 13:01

score 1 · Answer 3 · answered Mar 19 '21 at 12:11

1

This is an open issue at GitLab

Artifacts Filename Cannot Be Set with Dynamic Variables

I had a project variable defining the path to a zip file in a script which I reused at artifacts:paths level. The linked issue would have been more obvious had the artifacts:paths instance completely failed to get assigned but in my case it inherited a different value from that a mere two lines above in my job!

answered Mar 19 '21 at 12:11

Matt Bracewell

159
1
9

I don't see this will help anything because it addresses `artifacts:name` but not `artifacts:paths` . paths is still not dynamically usable as you can see in https://docs.gitlab.com/ee/ci/variables/where_variables_can_be_used.html – childno͡.de Mar 13 '23 at 15:32
@childno͡.de you're right! – Matt Bracewell Mar 14 '23 at 14:29

dynamically setting artifact path/folder structure on gitlab-ci.yml

3 Answers3

Linked