Hudson/Jenkins notification only on consecutive failures

Question

I am using Jenkins for running Selenium tests in a monitoring way. The failures are often a temporary phenomenon (timeouts etc.). I escalate projects' execution upon a failure with Naginator plugin and the next build usually passes.

Thus, I am looking for a possibility to use a failure count which would enable sending a notification only when test fails n consecutive times. Do you have any idea how can one do it?

score 5 · Accepted Answer · edited May 23 '17 at 12:07

Daniel, I believe your best bet is to script it yourself. I had a similar problem, and came up with a three-line solution on my own without any experience with Java/Groovy.

First off, you need a way to determine that a build has failed. See my problem for the solution.

Second, you need to store the number of failed builds somewhere. The file in the project workspace is the obvious location. Use this snippet as a base:

def f = new File(manager.build.getWorkspace().getRemote() + '/GroovyFailedBuildsCount.txt')
f.createNewFile()
f.write(text)

And third, you need to send an email. Off the top of my head you could mark the first failed builds as unstable, and when the limit is reached, mark the build as failed, and have the email-ext plugin to send email notifications only on failed builds.

Groovy getting started guide has been a great help for me.

bishop · Answer 2 · 2017-06-19T16:38:25.427

1

An alternative implementation:

Use the Disable Job plugin to disable the monitored job after a certain number of failures.
Create a monitoring job that checks if the monitored job is disabled. If the job is disabled, then fail the monitoring build and send an email.
(Optionally) Disable the monitoring job to keep it from sending you emails.

This has the happy side effect of also notifying you if someone manually disables your job.

To get a list of disabled jobs whose title contains "title filter":

curl -sS http://jenkins.example.com/api/json \
  | jq '.jobs[] | select(.name | contains("title filter")) | select(.color=="disabled") | .name'

edited Jun 19 '17 at 16:38

answered Dec 09 '16 at 14:59

bishop

37,830
11
104
139

For step 2, how should I create the monitoring job? I'm not seeing a built-in Jenkins project to do this. Does this require scripting? – Zac Jan 13 '17 at 16:15
@Zac Yes, a minor amount of scripting. You can use [Groovy](https://www.reddit.com/r/jenkinsci/comments/3nqa9s/how_to_check_if_another_job_is_disabled_using/) or you can use an [API call](http://stackoverflow.com/q/37571445/2908724). – bishop Jan 13 '17 at 16:46
thanks! That's what I figured. Will upvote your answer! – Zac Jan 13 '17 at 16:56

score 1 · Answer 3 · answered Aug 24 '17 at 14:20

Jenkins notification only on consecutive unstables

With Jenkins v. 2.65 and Groovy Postbuild Plugin v. 2.3.1

set your Jenkins notification plugin(mail/slack etc.) to send messages only if build failed.

Add this Groovy code as Job Postbuild Action

// debugging:
// manager.build.@result = hudson.model.Result.SUCCESS;

final int threshold = 2;

String jobName = manager.build.project.name;
def f = new File('/tmp/' + jobName + '_GroovyUnstableBuildsCount.txt');

// debugging:
// manager.createSummary("info.gif").appendText('path: ' + f.toString(), false, false, false, "red")

if(!f.exists()){ // file don't exists
    f.createNewFile();
}

// debugging:
// String fileContent = f.text;
// manager.createSummary("info.gif").appendText('fileContent: ' + fileContent, false, false, false, "red")


if (manager.build.result.isBetterOrEqualTo(hudson.model.Result.SUCCESS)) {
    f.write("0");

} else if (manager.build.result == hudson.model.Result.UNSTABLE) {

        int oldValue = 0;
        if(f.text != null && !f.text.empty && f.text != ''){
          oldValue = f.text.toInteger();
        }
    int newValue = oldValue + 1;

    if(newValue > threshold){ // trigger send message
        // set build to fail
        manager.build.setResult(hudson.model.Result.FAILURE);
        f.write("0"); // reset counter

    } else { // increase count

        f.write(newValue.toString());
    }
}

score 1 · Answer 4 · answered Oct 01 '19 at 11:46

You can use jenkins Editable Email Notification plugin and in advanced settings you will have an option of failure-X where you can specify after how many failures you need to send the email. Old versions of jenkins may not be helping with this. So you can use the following PRESEND SCRIPT in editable email-notification

try { //used try catch because NAGINATOR_COUNT will be a undefined property for first 
build
cancel = true
if ($NAGINATOR_COUNT && $NAGINATOR_COUNT == 2) { //where after 2nd retry it will 
trigger email
 cancel = false
}
} catch (e) {
 cancel = true
}

http://jenkins-ci.361315.n4.nabble.com/Email-only-after-all-retries-fail-td4713630.html

score 0 · Answer 5 · answered Apr 04 '12 at 02:41

0

The email-ext plugin allows you to email when the build status is "Failed" for two or more builds. Unfortunately it will email you every time after that. A groovy script might still be thew

answered Apr 04 '12 at 02:41

Michael

2,825
3
24
30

Hudson/Jenkins notification only on consecutive failures

5 Answers5