Gitlab-Ci. Transfer maven m2 local repository between stages

Question

Good day collegues. I use gitlab ci in production. I have a lot of stages. 1)Build artifact 2)Deploy to external server 3)Using jfrog cli to deploy to artifactory

I have a problem with caching maven local repository. My runners download all dependicies in first step(build) and do the same in the last step(deploy to artifactory). Also my runner delete all data from m2 folder before final stage:

Removing .m2/antlr/
Removing .m2/aopalliance/
Removing .m2/asm/
Removing .m2/avalon-framework/
Removing .m2/backport-util-concurrent/
Removing .m2/ch/
Removing .m2/classworlds/
Removing .m2/com/
Removing .m2/commons-beanutils/
Removing .m2/commons-chain/
Removing .m2/commons-cli/
Removing .m2/commons-codec/
Removing .m2/commons-collections/
Removing .m2/commons-digester/
Removing .m2/commons-io/
Removing .m2/commons-lang/
Removing .m2/commons-logging/
Removing .m2/commons-validator/
Removing .m2/dom4j/
Removing .m2/io/
Removing .m2/javax/
Removing .m2/junit/
Removing .m2/log4j/
Removing .m2/logkit/
Removing .m2/net/
Removing .m2/org/
Removing .m2/oro/
Removing .m2/sslext/
Removing .m2/xml-apis/
Removing .m2/xmlunit/
Removing jfrog
Removing target/

my gitlav-ci yaml(withou second step):

stages:
- build
- deploy-artifactory

variables:
  MAVEN_OPTS: "-Dmaven.repo.local=${CI_PROJECT_DIR}/.m2"
  MAVEN_CLI_OPTS: "-s .m2/settings.xml --batch-mode"

cache:
  key: "$CI_JOB_NAME"
  paths:
  - .m2/

build:
  image: maven:latest
  stage: build
  tags:
  - build
  script:
  - adduser --disabled-password --gecos '' mynonrootuser
  - chmod --recursive 777 .
  - git version
  - su --command='mvn versions:set -DgenerateBackupPoms=false -DnewVersion="$CI_COMMIT_REF_SLUG-SNAPSHOT"' mynonrootuser
  - su --command='mvn $MAVEN_CLI_OPTS clean install -B -f ./pom.xml' mynonrootuser
  artifacts:
    expire_in: 5 mins
    paths:
    - target/*.jar

  only:
  - develop

deploy-artifactory-snapshot:
  retry: 2
  image: maven:latest
  stage: deploy-artifactory
  tags:
  - release
  before_script:
  -  curl -fL https://getcli.jfrog.io | sh
  - ./jfrog rt config --url=${ARTIFACTORY_URL} --user=${ARTIFACTORY_USER} --password=${ARTIFACTORY_PASSWORD}
  - ./jfrog rt c show
  - export M2_HOME=/usr/share/maven
  - sed -i 's,MAVEN_REPO_SNAPSHOT_DEPLOYER,'"$MAVEN_REPO_SNAPSHOT_DEPLOYER"',g' configuration.yml
  - sed -i 's,MAVEN_REPO_RELEASES_DEPLOYER,'"$MAVEN_REPO_RELEASES_DEPLOYER"',g' configuration.yml
  - sed -i 's,MAVEN_REPO_SNAPSHOT_RESOLVER,'"$MAVEN_REPO_SNAPSHOT_RESOLVER"',g' configuration.yml
  - sed -i 's,MAVEN_REPO_RELEASES_RESOLVER,'"$MAVEN_REPO_RELEASES_RESOLVER"',g' configuration.yml
  script:
  - ./jfrog rt mvn "versions:set -DgenerateBackupPoms=false -Dartifactory.publish.artifacts=false -DnewVersion="$CI_COMMIT_REF_SLUG-SNAPSHOT"" configuration.yml --build-name=scdfrestrunner --build-number=$CI_JOB_ID
  - ./jfrog rt mvn "clean install" configuration.yml --build-name=scdfrestrunner --build-number=$CI_JOB_ID
  - ./jfrog rt bce scdfrestrunner $CI_JOB_ID
  - ./jfrog rt bp scdfrestrunner $CI_JOB_ID

  only:
  - develop

score 4 · Accepted Answer · answered Oct 27 '18 at 06:06

There are several ways to handle dependencies between build jobs/stages.

caching, guide is available here.

e.g cache section for m2

# Cache modules in between jobs
cache:
  key: ${CI_COMMIT_REF_SLUG} # cache is for per branch
  paths:
  - ${CI_PROJECT_DIR}/.m2

caching also can be carried out using something like minio (open source s3 like storage solution). Integration with gitlab guide is available here. But since caching requires zipping and unzipping large amounts of artifacts may introduce slowness (but obviously way speed than without cache). To avoid that problem docker volumes can be used.

docker volume

basically we can use a out of the box docker volume say ~/.m2 mounted from host machine location. If you using in kubernetes cluster you can leverage kubnernets volume solutions for this. You have to customize your build image bit for support mounting dependency dir, following is an example of mvn build image. This approach is better because you can't cache outside of your build dir.

e.g. docker image

FROM maven:3.5.4-jdk-8
# m2 dir on docker container
ENV MAVEN_OPTS "-Dmaven.repo.local=/.m2/repository"
ENV MAVEN_CLI_OPTS "-s /usr/local/.m2/settings.xml --batch-mode"

ADD .m2 /usr/local/.m2 # copy settings.xml to build image
# same as the m2 dir specified in MAVEN_OPTS
VOLUME /.m2

Artifacts

If you just want to pass your build dir e.g. /target, you can use gitlab artifacts solution, guide is available here.

I know this solutions. They don't work with my second step where i use jfrog cli for deploy. I suppose jfrog cli mvn commands coudn't see m2 folder. — Александр Шаповалов, Oct 29 '18 at 06:09
Ok, have you tried with `/target` without `*.jar` filtering? Also your `expire_in` is 5mins, make sure that doesn't exceed when its running deploy. — Ruwanka De Silva, Oct 29 '18 at 06:25
Sorry for my language if my explanations is not clear. I don't need a target directory. I need dependencies from m2 folder, from first step where i do build. I do not want my last step download it againt for jfrog cli mvn "clean install" action. I use global cache action to transfer this dependecies across project. — Александр Шаповалов, Oct 29 '18 at 08:30

Gitlab-Ci. Transfer maven m2 local repository between stages

1 Answers1