78

I have a Jenkinsfile with multiple stages and one of them is in fact another job (the deploy one) which can fail in some cases.

I know that I can made prompts using Jenkinsfile but I don't really know how to implement a retry mechanism for this job.

I want to be able to click on the failed stage and choose to retry it. jenkins-pipelines-with-stages

sorin
  • 161,544
  • 178
  • 535
  • 806
  • 2
    This overall feature request is at [JENKINS-33846](https://issues.jenkins-ci.org/browse/JENKINS-33846). It (dissapointingly) is only selected for declarative pipelines in [JENKINS-45455](https://issues.jenkins-ci.org/browse/JENKINS-45455). – mkobit Dec 11 '17 at 16:05

3 Answers3

57

You should be able to combine retry + input to do that Something like that

stage('deploy-test') {
   try {
     build 'yourJob'
   } catch(error) {
     echo "First build failed, let's retry if accepted"
     retry(2) {
        input "Retry the job ?"
        build 'yourJob'
     }
   }
}

you could also use timeout for the input if you want it to finish if nobody validates. There is also waitUntil that might be useful but i haven't used it yet

Edit : WaitUntil seems definitely the best, you should play with it a bit but something like that is cleaner :

stage('deploy-test') {
   waitUntil {
     try {
       build 'yourJob'
     } catch(error) {
        input "Retry the job ?"
        false
     }
   }
}

By the way, there is doc all of the steps here https://jenkins.io/doc/pipeline/steps

Chadi
  • 737
  • 1
  • 6
  • 21
fchaillou
  • 626
  • 6
  • 5
  • Is it going to add a retry prompt? I doubt. – sorin May 05 '16 at 17:58
  • Oh no you're right. i'll update my answer for that ! – fchaillou May 05 '16 at 19:05
  • 1
    Is it possible to enable the timeout only for the retry part? I may want to have a different timeout for the job. I didn't accepted the answer yet as I don't find a blocking job as a good solution. Ideally the retry option should be after the job already finished. Imagine that this job is triggered by a GitHub hook on a PR. I would prefer to see the failure on GitHub instead of no-answer in case of an error. – sorin Oct 03 '16 at 14:59
  • In my testing with waitUntil{} with the build pipeline step, I found I needed to explicitly return true, because that step doesn't return a boolean type. – Slack Flag Jun 05 '17 at 19:45
  • 1
    no longer works as posted, with error: Unknown stage section "waitUntil". Starting with version 0.5, steps in a stage must be in a ‘steps’ block. – Peter McIntyre Jan 15 '21 at 03:02
  • @fchaillou Here in the 1st code snippet, when the job fails for the 1st time, it will be retried 2 times no matter the job passes in 2nd go or not, right? Is it possible to stop retrying just after the job passes? – blackreaper Aug 29 '22 at 08:00
14

This one with a nice incremental wait

stage('deploy-test') {
 def retryAttempt = 0
 retry(2) {
    if (retryAttempt > 0) {
       sleep(1000 * 2 + 2000 * retryAttempt)
    }

    retryAttempt = retryAttempt + 1
    input "Retry the job ?"
    build 'yourJob'
 }
}
LuisKarlos
  • 527
  • 6
  • 6
  • 2
    `sleep()` has [default units of seconds](https://www.jenkins.io/doc/pipeline/steps/workflow-basic-steps/#sleep-sleep), so unless you want the first wait to be over an hour, specify `sleep(..., unit:"MILLISECONDS")` or use less seconds. – Carl Walsh Apr 07 '21 at 00:56
  • 1
    I think you can no longer put a `retry` at the top of a `stage` block: `Expected one of "steps", "stages", or "parallel" for stage "Code Coverage" @ line 423, column 17.` – jcollum Apr 15 '21 at 18:59
  • 1
    It only worked for me if I put it after the `steps` declaration. – jcollum Apr 15 '21 at 20:03
  • 1
    I believe you can put it outside of steps if you use `stage('test') { options { retry(2) } steps { echo "hello" } }` for a **declarative** pipeline only – Chris Maggiulli Aug 31 '21 at 17:46
10

This gist (not mine) was one of the better options that I found while trying to implement this functionality too. https://gist.github.com/beercan1989/b66b7643b48434f5bdf7e1c87094acb9

Changed it to a method in a shared library that just did retry or abort for my needs. Also added a max retries and made the timeout variable so that we could change it depending on the job or stage that needs it.

package com.foo.bar.jenkins

def class PipelineHelper {
    def steps

    PipelineHelper(steps) {
        this.steps = steps
    }

    void retryOrAbort(final Closure<?> action, int maxAttempts, int timeoutSeconds, final int count = 0) {
        steps.echo "Trying action, attempt count is: ${count}"
        try {
            action.call();
        } catch (final exception) {
            steps.echo "${exception.toString()}"
            steps.timeout(time: timeoutSeconds, unit: 'SECONDS') {
                def userChoice = false
                try {
                    userChoice = steps.input(message: 'Retry?', ok: 'Ok', parameters: [
                            [$class: 'BooleanParameterDefinition', defaultValue: true, description: '', name: 'Check to retry from failed stage']])
                } catch (org.jenkinsci.plugins.workflow.steps.FlowInterruptedException e) {
                    userChoice = false
                }
                if (userChoice) {
                    if (count <= maxAttempts) {
                        steps.echo "Retrying from failed stage."
                        return retryOrAbort(action, maxAttempts, timeoutMinutes, count + 1)
                    } else {
                        steps.echo "Max attempts reached. Will not retry."
                        throw exception
                    }
                } else {
                    steps.echo 'Aborting'
                    throw exception;
                }
            }
        }
    }
}

Example usage with a max of 2 retries that waits for 60s for input.

def pipelineHelper = new PipelineHelper(this)

stage ('Retry Example'){
    pipelineHelper.retryOrAbort({
        node{
            echo 'Here is an example'
            throw new RuntimeException('This example will fail.')
        }
    }, 2, 60)
}

Just remember to put nodes inside of the closure so that waiting for an input doesn't block an executor.

If you have the paid jenkins enterprise Cloudbees has a Checkpoint plugin that can better handle this, but it is not planned to be release for open source Jenkins (JENKINS-33846).

dben1713
  • 101
  • 2
  • 4