Is it possible to create parallel Jenkins Declarative Pipeline stages in a loop?

Question

I have a list of long running Gradle tasks on different sub projects in my project. I would like to run these in parallel using Jenkins declarative pipeline.

I was hoping something like this might work:

projects = [":a", ":b", ":c"]

pipeline {
    stage("Deploy"){
        parallel {
             for(project in projects){
               stage(project ) {
                   when {
                       expression {
                            someConditionalFunction(project)
                       }
                   }
                   steps {
                       sh "./gradlew ${project}:someLongrunningGradleTask"
                  }
                }   
             }
        }
    }
}

Needless to say that gives a compile error since it was expecting stage instead of for. Any ideas on how to overcome this? Thanks

score 61 · Answer 1 · edited Jun 01 '18 at 14:42

61

I was trying to reduce duplicated code in my existing Jenkinsfile using declarative pipeline syntax. Finally I was able to wrap my head around the difference between scripted and declarative syntax.

It is possible to use scripted pipeline syntax in a declarative pipeline by wrapping it with a script {} block.

Check out my example below: you will see that all three parallel stages finish at the same time after waking up from the sleep command.

def jobs = ["JobA", "JobB", "JobC"]

def parallelStagesMap = jobs.collectEntries {
    ["${it}" : generateStage(it)]
}

def generateStage(job) {
    return {
        stage("stage: ${job}") {
                echo "This is ${job}."
                sh script: "sleep 15"
        }
    }
}

pipeline {
    agent any

    stages {
        stage('non-parallel stage') {
            steps {
                echo 'This stage will be executed first.'
            }
        }

        stage('parallel stage') {
            steps {
                script {
                    parallel parallelStagesMap
                }
            }
        }
    }
}

edited Jun 01 '18 at 14:42

mkobit

43,979
12
156
150

answered Jan 24 '18 at 11:41

Max

852
1
9
19

@mkobit I see with Declarative Pipeline 1.2 I can add pipeline code to Shared library, but I cannot add method to under def call How can I add all above code in Shared Library including def part – user2661518 Aug 28 '18 at 16:44
1

I added `pipeline` part into Shared Library and I'm calling it like ```@Library('pipeline_lib') _ parallelDeclare()``` . But I get error ```groovy.lang.MissingPropertyException: No such property: parallelStagesMap for class: parallelDeclare``` – user2661518 Aug 28 '18 at 16:46
I would add more to this answer. How to throttle parallel stages: https://issues.jenkins-ci.org/browse/JENKINS-44085?focusedCommentId=354852&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-354852 – kivagant Nov 29 '18 at 11:44
though there are parallel jobs running,i see only 1 job that actually run and gives the output and other jobs that are created in parallel just waits. But on a 'ps -ef' i see multiple jobs running in parallel but internally waiting for 1 job (the last job in parallelStageMap) to complete. Should i use throttling to make all the parallel job run the actual task ? – Senthil A Kumar Feb 15 '19 at 10:16
6

for me this uses scripted pipeline stages; when you add agent, when, tools, or similar declarative constructs into generateStage it fails :( – Christian Mar 05 '19 at 14:46
1

This article successfully uses declarative stages inside parallel with pipelines 1.2, still it remains unclear how to generate stages programmatically: https://jenkins.io/blog/2017/09/25/declarative-1/ – Christian Mar 05 '19 at 14:47
@Christian did you find a solution to this issue ? – jhagege Sep 04 '19 at 08:53
No, I still use the imperative parallel approach. – Christian Sep 05 '19 at 14:40
]It runs each Job in different branch !? It translates the key of the Map entry ("${it}" ) as the branch name, and the value of the Map entry (generateStage(it)) as the stage to run for that branch. – Abdennour TOUMI Sep 15 '19 at 20:09
@Christian any news about "generate stages programmatically" ?? – Abdennour TOUMI Sep 15 '19 at 20:12
@AbdennourTOUMI, no, already reported at Sep 5th that I still use the imperative approach :( – Christian Sep 16 '19 at 09:06
Instead of making `parallelStagesMap` a map, is there way to make it a function that *returns* a map? I tried it, but when the build runs, the stage fails without any error logs. :( – Cameron Hudson Feb 06 '20 at 21:36
Works like a charm, in the declarative pipeline! Thank you!! – hilaf Oct 29 '20 at 14:36

score 6 · Answer 2 · answered Oct 23 '17 at 16:45

6

Parallel wants a map structure. You are doing this a little inside-out. Build your map and then just pass it to parallel, rather than trying to iterate inside parallel.

Option 2 on this page shows you a way to do something similar to what you are trying.

At this link you can find a complex way I did this similar to a matrix/multi-config job:

answered Oct 23 '17 at 16:45

Rob Hales

5,123
1
21
33

Cheers for the response and links Rob. Is that possible for the declarative pipeline too? Im using the syntax here: https://jenkins.io/doc/book/pipeline/syntax/#parallel It may be the case that its only possible using scripted pipeline – Frank Farrell Oct 23 '17 at 16:58
1

You would probably need to create the map in a script{} block or outside of the pipeline{} block, but yes, you can still do the same thing in declarative. You just have to combine it with a little scripted. The two merge together really well once you understand the variable scoping. – Rob Hales Oct 23 '17 at 17:59
@RobHales I have similar question but couldn't exactly figure it out can you please give a quick look https://stackoverflow.com/questions/52086102/jenkins-declarative-pipeline-1-3-run-parallel-for-all-jobs – user2661518 Aug 29 '18 at 23:32

Is it possible to create parallel Jenkins Declarative Pipeline stages in a loop?

2 Answers2

Linked