0

I've configured a pipeline for maven/java/spring project using a local gitlab-runner in my Mac. My project is hosted on gitlab.com. At every commit, it always takes forever, because it downloads all the dependencies.

Using the shared gitlab-runner the build and test is faster, but it uses my quota minutes. How can I improve it using local gitlab-runner? I can't wait 1 hour at every run.. Thanks in advance!

Preparing the "docker" executor
00:22
Using Docker executor with image maven:3.8.5-openjdk-17-slim ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Using docker image sha256:7b7c9f4ae103441d225d2b3ea4a5dd0f8cb5673b44ec984353853b2791099bde for docker:dind with digest docker@sha256:999fc127a51b8a86593ff9ba2518f14cbd18555849f8927fd56fa82395effe16 ...
Waiting for services to be up and running (timeout 30 seconds)...
Pulling docker image maven:3.8.5-openjdk-17-slim ...
Using docker image sha256:7337649a0aedd0916ce8bc19a3e1d58779afec34e06ec893fec37fde07661d94 for maven:3.8.5-openjdk-17-slim with digest maven@sha256:502e781d39f0b40fbd02eb23f5b7663618b76ba52034da218c64e92f6c5647be ...
Preparing environment
00:00
Running on runner-v5bnsne3-project-37118324-concurrent-0 via giuseppefalco-QT0Q7VN4KR...
Getting source from Git repository
00:02
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/gfalco77/maverick/.git/
Checking out 82e8826f as master...
Removing .m2/
Removing .sonar/
Removing target/
Skipping Git submodules setup
Restoring cache
00:00
Checking cache for master-1-protected...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
Successfully extracted cache
Executing "step_script" stage of the job script
Using docker image sha256:7337649a0aedd0916ce8bc19a3e1d58779afec34e06ec893fec37fde07661d94 for maven:3.8.5-openjdk-17-slim with digest maven@sha256:502e781d39f0b40fbd02eb23f5b7663618b76ba52034da218c64e92f6c5647be ...
$ echo "Building $MODULE"
Building 
$ mvn $MAVEN_CLI_OPTS -T 4 compile
Apache Maven 3.8.5 (3599d3414f046de2324203b78ddcf9b5e4388aa0)
Maven home: /usr/share/maven
Java version: 17.0.2, vendor: Oracle Corporation, runtime: /usr/local/openjdk-17
Default locale: en, platform encoding: UTF-8
OS name: "linux", version: "5.10.124-linuxkit", arch: "aarch64", family: "unix"
[INFO] Error stacktraces are turned on.
[INFO] Scanning for projects...
[INFO] Downloading from spring-milestones: https://repo.spring.io/milestone/com/gfs/phoenix/1.32/phoenix-1.32.pom
[INFO] Downloading from gitlab-maven: https://gitlab.com/api/v4/projects/37118324/packages/maven/com/gfs/phoenix/1.32/phoenix-1.32.pom

At the end

[INFO] Compiling 18 source files to /builds/gfalco77/maverick/target/classes
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  07:09 min (Wall Clock)
[INFO] Finished at: 2022-10-16T10:37:49Z
[INFO] ------------------------------------------------------------------------
Saving cache for successful job
00:02
Creating cache master-1-protected...
.m2/repository/: found 4492 matching files and directories 
No URL provided, cache will not be uploaded to shared cache server. Cache will be stored only locally. 
Created cache
Cleaning up project directory and file based variables
00:00
Job succeeded

Same for the other jobs , release, test - docker push - deploy

My gitlab-pipeline is (just showing build and test jobs)

variables:
  MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
  MAVEN_CLI_OPTS: "-s settings.xml --batch-mode --errors --fail-at-end --show-version"
  MAVEN_IMAGE: maven:3.8.5-openjdk-17-slim
  IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
  DOCKER_HOST: "tcp://docker:2375"
  DOCKER_DRIVER: "overlay2"
  DOCKER_TLS_CERTDIR: ""
  TESTCONTAINERS_RYUK_DISABLED: "true"

services:
  - docker:dind

stages:
  - build
  - test
  - release
  - docker-build
  - deploy

cache:
  paths:
    - .m2/repository/
  key: "$CI_BUILD_REF_NAME"

build-job:
  image: $MAVEN_IMAGE
  stage: build
  tags:
    - local-runner-maverick
  script:
    - echo "Building $MODULE"
    - mvn $MAVEN_CLI_OPTS -T 4 compile
  only:
    - merge_request
    - master

test-job:
  image: $MAVEN_IMAGE
  stage: test
  tags:
    - local-runner-maverick
  variables:
    SONAR_USER_HOME: "${CI_PROJECT_DIR}/.sonar"  # Defines the location of the analysis task cache
    GIT_DEPTH: "0"  # Tells git to fetch all the branches of the project, required by the analysis task
  cache:
    key: "${CI_JOB_NAME}"
    paths:
      - .sonar/cache
  script:
    - echo "Testing $MODULE"
    - mvn $MAVEN_CLI_OPTS verify -e sonar:sonar -Dsonar.qualitygate.wait=true -Dsonar.projectKey=gfalco77_maverick
  allow_failure: false
  only:
    - merge_requests
    - master
 

I'm using JIB to build the image and this is the maven plugin configuration

<plugin>
    <groupId>com.google.cloud.tools</groupId>
    <artifactId>jib-maven-plugin</artifactId>
    <version>${google.jib.version}</version>
    <configuration>
        <container>
            <creationTime>USE_CURRENT_TIMESTAMP</creationTime>
        </container>
        <from>
            <image>openjdk:17-jdk-slim-buster</image>
        </from>
    </configuration>
    <executions>
        <execution>
            <id>Skip-build-on-release</id>
            <phase>none</phase>
        </execution>
    </executions>
</plugin>

Finally this is the config.toml

[[runners]]
  name = "local-runner-maverick"
  url = "https://gitlab.com/"
  token = "..."
  executor = "docker"
  environment = ["DOCKER_DRIVER=overlay2"]
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "maven:3.8.5-openjdk-17-slim"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/certs/client", "/cache", "/.m2"]
    shm_size = 0
Gfalco
  • 77
  • 3
  • 13
  • You are using a cache based on your branch/tag name, do you have this issue with all commits on the same branch, or only when you make new branches or tags? essentially all your maven stuff should be cached unless its put somewhere else – Chris Doyle Oct 16 '22 at 13:52
  • I think after new branch, isn't there anything to speed it up? I have a pipeline on merge request and master – Gfalco Oct 16 '22 at 16:11
  • You are specifically tying your cache to the branch/tag. So when you make a new branch or push to master they wont be able to use that cache. You can read more here. If you want all pipelines in the project to share the cache then you could use the project ID or something as the cache. see more here https://docs.gitlab.com/ee/ci/yaml/index.html#cachekey – Chris Doyle Oct 16 '22 at 17:27

0 Answers0