How to make GitLab Windows shared runners to build faster?

Question

Background

I have a CI pipeline for a C++ library I've been developing. So far, I can distribute this lib to Linux and Windows systems. Since I use GitLab to build, test and package my lib, I'd like to have my Windows builds running faster and I have no clue on how to do that.

Currently, I use the following script for my Windows builds:

.windows_template:
tags:
    - windows
before_script:
    - choco install cmake.install -y --installargs '"ADD_CMAKE_TO_PATH=System"'
    - choco install python --pre -y
    - choco install git -y
    - $env:ChocolateyInstall = Convert-Path "$((Get-Command choco).Path)\..\.."; Import-Module "$env:ChocolateyInstall\helpers\chocolateyProfile.psm1"; refreshenv
    - python -m pip install --upgrade pip
    - pip install conan monotonic

The problem

Any build with the script above can take up to 10 minutes; worse: I have two stages, each one taking the same amount of time. This means that my whole CI pipeline will take 20 minutes to finish because of slowness in Windows builds.

Ideal solution

EVERYTHING in my before_script can be cached or stored as an image. I only need some hints on how to do it properly.

Additional information

I use the following tools for my builds:

• CMake: to support my building process;
• Python3: to test and build packages;
• Conan (requires Python3): to support the creation of packages with several features, as well as distribute them;
• Git: to download Googletest in CMake configuration step This is already provided in the cookbooks - I might just remove this extra installation step in my before_script;
• Googletest (requires Python3): testing library;
• Visual Studio DEV Tools: to compile the library This is already in the cookbooks.

Answer 1

Installing packages like this (whether it's OS packages though apt-get install... or pip, or anything else) is generally against best practices for CI/CD jobs because every job that runs will have to do the same thing, costing a lot of time as you run more pipelines, as you've seen already.

A few alternatives are to search for an existing image that has everything you need (possible but not likely with more dependencies), split up your job into pieces that might be solved by an image with just one or two dependencies, or create a custom docker image to use in your jobs. I answered a similar question with an example a few weeks ago here: "Unable to locate package git" when running GitLab CI/CD pipeline

But here's an example Dockerfile with Windows:

# Dockerfile
FROM mcr.microsoft.com/windows
RUN ./install_chocolatey.sh
RUN choco install cmake.install -y --installargs '"ADD_CMAKE_TO_PATH=System"'
RUN choco install python --pre -y
RUN choco install git -y
...

The FROM line says that our new image extends the mcr.microsoft.com/windows base image. You can extend any image you have access to, even if it already extends another image (in fact, that's how most images work: they start with something small, like a base OS installation, then add things needed for that package. PHP for example starts on an Ubuntu image, then installs the necessary PHP packages).

The first RUN line is just an example. I'm not a Windows user and don't have experience installing Chocolatey, but you'd do here whatever you'd normally do to install it locally. The rest are for installing whatever else you need.

Then run

docker build /path/to/dockerfile-dir -t mygroup/mytag:version

The path you supply needs to be the directory that contains the Dockerfile, not the Dockerfile itself. The -t flag sets the image's tag after it's built (though you can do that with a separate command, docker tag too).

Next, you'll have to log into whichever registry you're using (Docker Hub (https://docs.docker.com/docker-hub/repos/), Gitlab Container Registry (https://docs.gitlab.com/ee/user/packages/container_registry/), a private registry your employer may support, or any other option.

docker login my.docker.hub.com

Now you can push the image to the registry:

docker push my.docker.hub.com/mygroup/mytag:version

You'll have to review the information in the docs about telling your Gitlab runner or pipelines how to authenticate with the registry (unless it's Public on Docker Hub or you use the Gitlab Container Registry) https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#define-an-image-from-a-private-container-registry

Once all that's done, you can use your new image in your CI jobs, and everything we put into the image will be ready to use:

.windows_template:
image: my.docker.hub.com/mygroup/mytag:version
tags:
    - windows
...