GitLabRunner CI very slow on Windows 10

Question

I have GitLabRunner 14.4.0 (4b9e985a) installed as a service on Windows Server with follow parameters:

Sorry for non English, but I guess it is clear what is going there.

Nothing is run on the server except GitLab-Runner, so all resources are free.

Here is the config.toml:

concurrent = 5
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "TorgovayaSystema"
  url = "https://srvgitlab.maxi-net.ru/"
  token = "xxxxxx_xxxxxxxxxxxxx"
  executor = "shell"
  shell = "cmd"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]

I tried both shells - cmd and pwsh.

I run task from follow CI config:

stages:
  - build

build-modules-job:
  stage: build
  variables: 
    ErrorActionPreference: stop
  script:
    - .\gradlew.bat compile
  tags:
    - cpp

When I run gradlew.bat compile separately on this machine, it takes one minute, but inside the runner in goes for timeout, it was one hour before.

So, my problem is 1 hour timeout

I tried to use -d flag, and get many:

2021-11-12T18:21:33.777+0300 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
2021-11-12T18:21:33.777+0300 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
2021-11-12T18:21:33.777+0300 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
2021-11-12T18:21:33.778+0300 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
2021-11-12T18:21:33.778+0300 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
2021-11-12T18:21:33.778+0300 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
2021-11-12T18:21:43.777+0300 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
2021-11-12T18:21:43.777+0300 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
2021-11-12T18:21:43.778+0300 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
2021-11-12T18:21:43.779+0300 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
2021-11-12T18:21:43.779+0300 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
2021-11-12T18:21:43.780+0300 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.

I guess it is not a problem in gradle or CI configuration, thus git initialization take much time too...
I guess it is something in gitlab-runner.exe...
Have someone ideas?

What you're seeing is related to the gradle daemon. Have you tried running gradle without the daemon and does that resolve your issue? If so, you're likely running into a permissions issue; GitLab's runner uses a non-privileged user on your system, which is likely a different user than you're using to test. That may be causing issues. — Patrick, Nov 13 '21 at 19:55
Is there anything about your system configuration that may account for this, like a network proxy configuration? Also as Patrick states, check your user permissions on the user for the gitlab-runner. This may also help: https://stackoverflow.com/a/51128852/5747944 — sytech, Nov 14 '21 at 00:35
Thank you all for your answers. I tried with my own acount using --user and --password option, and --no-daemon also, so it solved part of the problems. — Olga Pshenichnikova, Nov 15 '21 at 12:31

Olga Pshenichnikova · Accepted Answer · 2021-11-15T13:53:11.540

OK, so first of all, I want to thank people who wrote comments under my post.

I actually tried both suggested ways:

Run gitlab-runner.exe with my own acount using gitlab-runner install --user user --password password option
Run gradle task --no-daemon

Also, very important, when you use gradle or maven inside batch scripts, you need to execute it with call statement, for exapmle:

Using call: call gradle compile --no-daemon

In such a way, the script will be continue.

But the system was still hanging up...

I tried to run all this in separate batch with compilation commands, which are made the same, the gradle plugin made for build the Borland CPP:

Using script with manual run of bpr2mak.exe and make.exe

Than I found the problem... When I run it from inside Gradle plugin, it used such a piece of code:

Process process = Runtime.getRuntime().exec(command, envVars.toArray(new String[0]), cwd);
BufferedReader out = new BufferedReader(new InputStreamReader(process.getInputStream()));

String line;
while ((line = out.readLine()) != null) {
    logger.log(line);
    System.out.println(line);
}

process.waitFor();
return process.exitValue();

So there is some specificity of Runtime.exec() and there is blocking read in out.readLine() with mutual block of process.waitFor(). You can read here about.

So here the process hunged up! It freezed over:

Loaded cached pre-compiled headers
DocAssImpl.cpp:
FmDocAss.cpp:

When I ADDED stderr BEFORE stdout, it continues, and gives me an expected message.

Whole code:

Process process = Runtime.getRuntime().exec(command, envVars.toArray(new String[0]), cwd);
BufferedReader out = new BufferedReader(new InputStreamReader(process.getInputStream()));
BufferedReader err = new BufferedReader(new InputStreamReader(process.getErrorStream()));

String line;
while ((line = err.readLine()) != null) {
    logger.log(line);
    System.out.println(line);
}

while ((line = out.readLine()) != null) {
    logger.log(line);
    System.out.println(line);
}

process.waitFor();
return process.exitValue();

Resulting messages;

BUILD FAILED in 3s
2 actionable tasks: 2 executed
Cleaning up project directory and file based variables 00:00
ERROR: Job failed: exit status 1

So after all, the problems were in include paths... Our build system widely uses subst, so for successfull complilation, it was necessary to add follow commands to the build scripts:

subst L: C:\Work\projects\Library\
subst P: C:\Work\projects\TradingSystem\

GitLabRunner CI very slow on Windows 10

1 Answers1