20

We use Jenkins for our CI build system. We also use 'concurrent builds' so that Jenkins will build each change independently. This means we often have 5 or 6 builds of the same job running simultaneously. To accommodate this, we have 4 slaves each with 12 executors.

The problem is that Jenkins doesn't really 'load balance' among its slaves. It tries to build a job on the same slave that it previously built on (presumably to reduce the time syncing from source control). This is a problem because Jenkins will build all 6 instances of our build on the same slave (or more likely between 2 slaves). One build machine gets bogged down and runs very slowly while the rest of them sit idle.

How do I configure the load balancing behavior of Jenkins, and how it controls its slaves?

Jeff
  • 3,829
  • 1
  • 31
  • 49
Jay Spang
  • 2,013
  • 7
  • 27
  • 32
  • 6
    Closed as off-topic, is that for real? I'm sorry to say I don't often write these types of comments, but -1 to the mods for that. The question clearly states a specific problem and is asking for a configuration solution. – Ed Randall Jul 18 '19 at 07:31

5 Answers5

22

We were facing a similar issue. So I've put together a plugin that changes the Load Balancer in Jenkins to select a node that currently has the least load - https://plugins.jenkins.io/leastload/

Any feedback is appreciated.

Eric Citaire
  • 4,355
  • 1
  • 29
  • 49
bstick12
  • 1,699
  • 12
  • 18
  • 1
    Great, thanks! The latest plugin build date is currently "Jun 27, 2013". Is it compatible with newer Jenkins versions, e.g. 1.625.3 ? – t0r0X Jan 15 '16 at 22:49
  • It should be I'm currently using it with 1.625.2 – bstick12 Jan 18 '16 at 15:52
  • I used this plugin but it didn't solve my problem. My jobs are in queue even when the node is available. https://stackoverflow.com/questions/58943134/jenkins-build-queue-very-small-jobs-getting-blocked – Psdet Nov 21 '19 at 16:26
3

If you do not find a plugin that does it automatically, here's an idea of what you can do:

  • Install Node Label Parameter plugin

  • Add SLAVE parameter to your jobs

  • Restrict jobs to run on ${SLAVE}

  • Add a trigger job that will do the following:

    • Analyze load distribution via a System Groovy Script and decide on which node to start next build.
    • Dispatch the build on that node with Parameterized Trigger plugin by assigning appropriate value to SLAVE parameter.

In order to analyze load distribution you need to install Groovy plugin and familiarize yourself with Jenkins Main Module API. Here are some useful initial pointers.

Raúl Salinas-Monteagudo
  • 3,412
  • 1
  • 24
  • 22
malenkiy_scot
  • 16,415
  • 6
  • 64
  • 87
  • There doesn't seem to be a plugin that does this (that I can find). While this suggestion is a lot more... 'involved' than just using a plugin, it looks like our best option! – Jay Spang Jun 21 '12 at 23:26
3

If your build machines cannot comfortably handle more than 1 build, why configure them with 12 executors? If that is indeed the case, you should reduce the number of executors to 1. My Jenkins has 30 slaves, each with 1 executor.

sti
  • 11,047
  • 1
  • 27
  • 27
  • 5
    Our Jenkins slaves handle many tasks, one of which is to compile our code. The slaves have 12 executors because it can do 12 things at once (the builds being the 'biggest' of these things). It can even run 4-5 builds just fine, it just slows things down (and we want the feedback of these builds to be as fast as possible). By reducing these slaves to 1 executor, we'd have 15 cores sitting idle the majority of the day, not a very efficient use of resources. – Jay Spang Jun 20 '12 at 18:12
  • I have jobs in queue but doesn't run on node which is available. How do you resolve that? – Psdet Nov 20 '19 at 20:50
  • This is a good idea, especially if your build system will try to use all available cores. In the latter case, trying to have 15 build processes trying to use 12 cores each will hog the machine indeed. – Raúl Salinas-Monteagudo Mar 22 '22 at 11:42
1

You may also use the Throttle Concurrent Builds plugin to restrict how many instances of a job can run in parallel on the same node

user1506978
  • 11
  • 1
0

I have two labels -- one for small tasks and one for big tasks. I have one executor for the big task and 4 for the small tasks. This does balance things a little.

jeremy mordkoff
  • 61
  • 6