How to prevent certain Jenkins jobs from running simultaneously?

Question

I have a couple of jobs that use a shared resource (database), which sometimes can cause builds to fail in the (rare) event that the jobs happen to get triggered simultaneously.

Given jobs A through E, for example, is there any way to specify that A and C should never be run concurrently?

Other than the aforementioned resource, the builds are independent of each other (not e.g. in a upstream/downstream relation).

A "brute-force" way would be limiting number of executors to one, but that obviously is less than ideal if most jobs could well be executed concurrently and there's no lack of computing resources on the build server.

@marc.guenther: John's comment was related to [Locks and Latches plugin](https://wiki.jenkins-ci.org/display/JENKINS/Locks+and+Latches+plugin) mentioned in [pwan's answer](http://stackoverflow.com/a/6279210/56285). It is [proposed for deprecation here](https://wiki.jenkins-ci.org/display/JENKINS/Proposed+Plugin+Deprecation), but other than that I don't see any signs of it going away. So I guess you can achieve this using either 'Locks and Latches' or 'Throttle Concurrent Builds'. — Jonik, Mar 18 '13 at 14:31
Avoid the Locks and Latches plugin. It is on the deprecation list for a reason. If you manually interrupt a job while it is waiting to acquire a lock, the job will not run anymore until the Jenkins master is restarted. The Throttle Concurrent Builds plugin is a much better option that does not compromise your Jenkins installation. — Fabian Ritzmann, Feb 03 '14 at 12:13

score 46 · Accepted Answer · edited Sep 09 '21 at 14:16

46

There are currently 2 ways of doing this:

Use the Throttle Concurrent Builds plugin.
Set up those jobs to run on a slave having only 1 executor.

edited Sep 09 '21 at 14:16

Joshua Wade

4,755
2
24
44

answered Mar 27 '12 at 20:56

sti

11,047
1
27
27

2

For some reason the simple approach of using only one executor didn't work for me. I configured it to 1, and 4 days later someone reported that there were 2 running,.. and the config was showing: max = 1. – sorin Mar 26 '13 at 19:20
Does someone know if it is possible to use the Throttle Concurrent Builds to throttle two pipelines but allow at the same time that one of the pipelines executes its own jobs concurrently? For example, I have a pipeline A and a pipeline B, I do have A running and B won't be able because A has some job running. I want to be able to have N jobs of A running while none of B running if there is any of A doing it. – Marc43 Dec 03 '20 at 12:30

score 14 · Answer 2 · edited May 23 '17 at 11:33

14

The Locks and Latches plugin here should help.

This question is probably a dupe of How do I ensure that only one of a certain category of job runs at once in Hudson?

edited May 23 '17 at 11:33

Community

1
1

answered Jun 08 '11 at 13:10

pwan

2,894
2
24
38

3

Thanks, got it working! Quick reference: 1) define locks in **Manage Jenkins -> Configure System -> Locks** (in my case I used the name of the shared database); 2) set relevant jobs to use the locks: **Configure -> Build Environment -> Locks** – Jonik Jun 08 '11 at 13:58
My question indeed seems to boil down to the same thing as the older one. But I'd argue that this one is much clearer (no wall of basically irrelevant text in question body & no misleading tags)... – Jonik Jun 08 '11 at 14:03
1

Just a heads up - sometimes multiple jobs get the lock at the same time. You can avoid this by limiting the number of executors on that node. – Feasoron Sep 16 '11 at 20:56
8

As @John T Dyer pointed out, Locks and Latches is on the [proposed plugin deprecation list](https://wiki.jenkins-ci.org/display/JENKINS/Proposed+Plugin+Deprecation). Reading the comments on that page, it seems **[Throttle Concurrent Builds](http://wiki.hudson-ci.org/display/HUDSON/Throttle+Concurrent+Builds+Plugin)** should be a viable replacement. – Jonik Dec 30 '11 at 22:47
1

For Jenkins: [Throttle Concurrent Builds plugin](https://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin) – Tuukka Mustonen May 04 '12 at 11:07

Konstantin A. Magg · Answer 3 · 2016-12-30T11:20:02.323

That's an old question, but the topic can still be relevant, especially when running application tests on Jenkins.

The Lockable Resources Plugin allows you to define lockable resources that can be used by builds. If your build requires an resource, it takes the lock. If a second build requires the same resource (which then is already locked), it will be queued for the resource to be free.

Although the docs use computers or printers as examples for lockable resources, the database example from above should work as well.

In opposite to the Locks and Latches Plugin mentioned in answers from 2012, this package seems to be currently maintained (currently ~2016).

score 2 · Answer 4 · answered Jul 24 '13 at 14:26

2

Have a look at the External Resource Dispatcher Jenkins plugin, which was first published in November 2012. This (relatively) new plugin seems to exactly cover this use case.

answered Jul 24 '13 at 14:26

MarnixKlooster ReinstateMonica

9,640
14
54
108

1

It's full of bugs (I had to modify the source to get something even half-working) and no longer works with the latest Jenkins. Other than that it would have been exactly what I wanted :-) – Cameron Jul 28 '17 at 15:13

score 0 · Answer 5 · answered Jun 03 '14 at 13:38

N.B. you don't need physical or virtual hardware for a slave/node, you can set up "slaves" that run on the master server.

Manage Jenkins > Manage Nodes > New node

and make a "dumb slaves" each with its own root directory.

Create a few slaves, execute them when the server boots, and then you have essentially created pools of executors.

You might have, say...

db - only one executor in your case. compile - limit according to hardware or # of CPUs. scripts - have many executors for all those little jobs that Jenkins is good at doing.

score 0 · Answer 6 · answered Mar 08 '16 at 03:02

Old question, and whether this will work for your application I can't be sure as you didn't mention details of your application. However, I wanted to add the way that I handled this in our Rails application test suite.

Our application's database configuration (database.yml) isn't in the source repository. Instead, it lives in /var/lib/configs/uniquing_database.yml on the VM which runs our Jenkins instance.

One of the steps of our build process involves copying this config file to the project workspace:

cp /var/lib/jenkins/configs/myapp_unique_database.yml config/database.yml

and that config takes workspace and build number information exposed to the environment by Jenkins into account in order to create a uniquely named database for that job and it's specific execution:

test:
  adapter: postgresql
  encoding: unicode
  host: 127.0.0.1
  port: 5432
  database: myapp_test<%= ENV['JOB_NAME'].split('/').last %><%= ENV['BUILD_NUMBER'] %>

The rest of our build proceeds without any knowledge or care that it's running in a distinct database. Finally, at the end of our build, we make sure to drop that database so we don't have a bunch of test databases polluting the file system:

RAILS_ENV=test bundle exec rake db:drop

How to prevent certain Jenkins jobs from running simultaneously?

6 Answers6

Linked