2

I would like to ask, how to configure GitHub workflow self-hosted runner to be able to pickup multiple tasks together. I'm running the GitHub self-hosted runner on Ubuntu 22.04 with docker.

Everything is working great, only still I must wait longer time, because runner is not able to pickup multiple tasks together although in workflow it's so set up. Also in GitHub in workflow screen, in summary section, I see, that some of tasks are able to run parallel, but runner pickup only one at time.

Thanks a lot for an each help :-)

klaucode
  • 180
  • 1
  • 10

1 Answers1

1

The runner process will only run a single job at a time. But if the host of the runner has enough CPU and IO and RAM to host more than one job at a time, you can always install a extra runner on the same VM in a different folder. There is no limit to the number of runners you can put on a single machine.

The only downside is that each runner will have its own workspace, task cache etc and that in order to bring the host offline, you should turn off all runners on the same host individually.

There are also a few downsides and security issues with such a setup, since one job could influence the outcome of another job. Ideally each runner runs under its own user and won't have access to any folder owned by the other runner. Also be careful with any jobs that install things globally on the server (e.g. npm install -g or apt-get install), two jobs trying this at the same time may run into eachothers locks.

Maybe a better way to leverage the full power of the host is to run multiple VMs or Containers.

jessehouwing
  • 106,458
  • 22
  • 256
  • 341
  • Thank you VERY MUCH for your answer. I was thinking, that I need only to find somewhere some configuration file and configure runner allow somewhere multiple runner threads (with similar way, like GitLab runner). If there is really not a "good practice" way, how to do that, therefore GitLab self-runner seems to be better without any complicated configuration. Now as you write, I must create a multiple users and run multiple instances of GitHub runner, I'm thinking correctly? – klaucode Mar 05 '23 at 19:09
  • 1
    The GitLab runner will suffer from the same issues of parallel jobs being able to influence each other. If you trust everyone on the server, you can take the risk. I've done it many times. But security incidents like Solargate happen by having a persistent build agent that allows one job to influence another. Ideally you'd use containers or temporary VMs to run one job and one job only, then reset their state. There is a long list of projects for GitHub Actions that support this scenario. https://github.com/jonico/awesome-runners – jessehouwing Mar 05 '23 at 19:26
  • Installing 2 agents on the same box is very easy to do. Just copy the downloaded runner archive, extract it to a new folder and run the configuration script. Done! – jessehouwing Mar 05 '23 at 19:27
  • ...yes, but I'm using runner together with docker. Workflow is running inside the docker containers. The host machine is dedicated for this. But I don't know, why GitLab runner is able to run parallel proceses without any problems and GitHub not. GitHub runner is somehow limited only to one process :-( – klaucode Mar 05 '23 at 20:20
  • ...but If I will need to be able to run for example 10 parallel jobs, I must manually copy and run the runner script 10 times? – klaucode Mar 05 '23 at 20:22
  • 1
    Yes. it's a design choice I guess. It's a more lightweight runner that won't have to synchronize access to its cached files and won't have to juggle local clones of git repos at the same time. The reduced size and complexity reduces bugs and improves security. But you'll have to install it 10 times. Or use one of those scaling solutions to automatically scale runners in docker or k8s or azure-scalesets or... – jessehouwing Mar 05 '23 at 20:41