0

I have some parallel stages in my Jenkins pipeline. They are all identical, except that they run on different agents:

stage {
    parallel {
        stage {
            agent {
                label 'agent-1'
            }
            steps {
                sh 'do task number 468'
            }
        }
        stage {
            agent {
                label 'agent-2'
            }
            steps {
                sh 'do task number 468'
            }
        }
        stage {
            agent {
                label 'agent-3'
            }
            steps {
                sh 'do task number 468'
            }
        }
    }
}

I want to add more parallel stages on more nodes, but the script is long and repetetive. What's the best way to rewrite this to tell jenkins to parallelize the same steps across agents 1, 2, 3, 4...etc?

cfrick
  • 35,203
  • 6
  • 56
  • 68
Seth Lutske
  • 9,154
  • 5
  • 29
  • 78
  • Check [sequential-stages-within-parallel-pipeline-in-jenkins](https://stackoverflow.com/questions/59688963/sequential-stages-within-parallel-pipeline-in-jenkins/59690399#59690399) – Sers Jul 02 '21 at 20:20
  • Can you glean anything from [Welxome to the Matrix](https://www.jenkins.io/blog/2019/11/22/welcome-to-the-matrix/) blog post? – Ian W Jul 03 '21 at 10:33

1 Answers1

1

Please see below code which will create and run the stage on multiple agents:

// Define your agents
def agents  = ['agent-1','agent-2','agent-3']

def createStage(label) {
    return {
        stage("Runs on ${label}") { 
            node(label) {
                // build steps that should happen on all nodes go here
                echo "Running on ${label}"
                sh 'do task number 468'
            }
        }
    }
}

def parallelStagesMap = agents.collectEntries {
    ["${it}" : createStage(it)]
}
pipeline {
    agent none
    stages {
         stage('parallel stage') {
            steps {
                script {
                    parallel parallelStagesMap
                }
            }
        }        
    }
}

More information is available at : Jenkins examples

Altaf
  • 2,838
  • 1
  • 16
  • 8
  • It looks promising, I'll give it a try when I can. Thank you! – Seth Lutske Jul 02 '21 at 21:14
  • Following this logic, I get an error `tput: No value for $TERM and no -T specified` in the Jenkins console. I checked out [this thread](https://stackoverflow.com/questions/29979966/tput-no-value-for-term-and-no-t-specified-error-logged-by-cron-process) on each node. I'm not sure what might be calling tput in your logic differently than writing out the stages manually. Any ideas for me? – Seth Lutske Jul 06 '21 at 15:30
  • @SethLutske I ran the same code in jenkins pipeline and it ran successfully.Maybe you can try setting ``TERM=dumb`` as mentioned in the thread. – Altaf Jul 06 '21 at 17:49
  • Is there a way to get the index of the entry and pass it as a second param to createStage? I want to be able to `echo` what # a stage is, but I can't seem to figure out how to do that with `collectEntries` – Seth Lutske Jul 06 '21 at 22:04
  • @SethLutske you can do it in below way : ``def parallelStagesMap = agents.collectEntries { def i=0 ["${it}" : createStage(it,i++)] }`` and you can use ``def createStage(label, idx)`` – Altaf Jul 07 '21 at 15:08
  • Thank you for help. This is basically working, though I will say that creating parallel stages in this fashion seems to be introducing bugs (and failures) that don't happen when writing out stages in parallel by hand. I'm not sure why that is, but your answer is a good answer, and my new bugs probably deserve their own questions. Thank you! – Seth Lutske Jul 07 '21 at 15:55
  • I've described the primary issue I'm having here - [Jenkins parallel stages - enoent ENOENT: no such file or directory ](https://stackoverflow.com/questions/68289958/jenkins-parallel-stages-enoent-enoent-no-such-file-or-directory). Perhaps you can take a look and see if you have some ideas? – Seth Lutske Jul 07 '21 at 16:40