Currying groovy CPS closure for parallel execution

Question

We do some dynamic creation of parallel steps in some of our jobs. Thanks to this thread I found how to dynamically create the map with parameters for use in the parallel step.

However now I wanted to reuse parts of the code which is used to create those parallel steps. For this I feel that I would need to curry the closures.

However currying seems not to work correctly. Referencing the loop variable (valueCopy) inside the closure does the right thing (as mentioned here) but currying does not do what I would expect.

Am I doing something wrong, is this just not (yet) supported, are there any workarounds? Is this probably a bug in Jenkins pipeline?

Hope anybody has a clue why this doesn't work and/or how to get it working.

Jenkins: LTS (2.32.1) & latest plugin updates as of 2017/01/19.

Solution:

After upgrading to Pipeline: Groovy plugin version 2.40 eveything is working as expected now.

Pipeline script executed:

def echoSome(val) {
    echo val
}

def buildClosures() {
    def someList = ["1", "2", "3"]
    def closures = [:]
    for (value in someList) {
        final valueCopy = value

        closures[value] = {val ->
                echo valueCopy.toString()
                echo val.toString()
            }.curry(value)
    }
    closures
}

parallel buildClosures()

Output:

[Pipeline] parallel
[Pipeline] [1] { (Branch: 1)
[Pipeline] [2] { (Branch: 2)
[Pipeline] [3] { (Branch: 3)
[Pipeline] [1] echo
[1] 1
[Pipeline] [1] echo
[1] 3
[Pipeline] [1] }
[Pipeline] [2] echo
[2] 2
[Pipeline] [2] echo
[2] 3
[Pipeline] [2] }
[Pipeline] [3] echo
[3] 3
[Pipeline] [3] echo
[3] 3
[Pipeline] [3] }
[Pipeline] // parallel
[Pipeline] End of Pipeline
Finished: SUCCESS

Expected Output:

[Pipeline] parallel
[Pipeline] [1] { (Branch: 1)
[Pipeline] [2] { (Branch: 2)
[Pipeline] [3] { (Branch: 3)
[Pipeline] [1] echo
[1] 1
[Pipeline] [1] echo
[1] 1
[Pipeline] [1] }
[Pipeline] [2] echo
[2] 2
[Pipeline] [2] echo
[2] 2
[Pipeline] [2] }
[Pipeline] [3] echo
[3] 3
[Pipeline] [3] echo
[3] 3
[Pipeline] [3] }
[Pipeline] // parallel
[Pipeline] End of Pipeline
Finished: SUCCESS

score 5 · Answer 1 · answered May 09 '17 at 13:07

I'm not sure if it's the currying, or the for loop, but this function needs to be marked as NonCPS as described here: https://github.com/jenkinsci/pipeline-examples/blob/master/docs/BEST_PRACTICES.md#groovy-gotchas

Essentially, do this:

@NonCPS
def buildClosures() {
    def someList = ["1", "2", "3"]
    def closures = [:]
    for (value in someList) {
        final valueCopy = value

        closures[value] = {val ->
                echo valueCopy.toString()
                echo val.toString()
            }.curry(value)
    }
    closures
}

I think it's your for loop, but regardless, anytime you don't use classic "C Style" loops, you'll need to mark your function as NonCPS.

Thanks for the answer. For some reason I was not notified :(. I'll give it a try as soon as possible. — Joerg S, Jun 21 '17 at 08:30
Marked as solved as this really solves my question. Unfortunately I missed to mention that of course I'd like to use CPS code inside the parallel closures :( — Joerg S, Jun 21 '17 at 08:39

score 0 · Answer 2 · answered Jun 09 '17 at 23:07

This seems to be a limitation of either the Groovy language or the Jenkins groovy runtime, I'm not sure which, but it's worth noting their examples do exactly as you have done, declaring a new variable for each iteration of the loop.

They have commented their example

// fresh variable per iteration; i will be mutated

I don't think that using C-style loops will remove this limitation and currying (which would be required for this use case) does not resolve the issue either. Clumsy, but easy enough to work around.

https://github.com/jenkinsci/pipeline-plugin/blob/master/TUTORIAL.md#creating-multiple-threads

score 0 · Accepted Answer · answered Sep 21 '17 at 07:56

0

Found that with the latest Pipeline: Groovy plugin (2.40) combined with at least Jenkins Version 2.60.3 (works although the plug-ins homepage states that you need at least Jenkins 2.73.3) everything works as expected.

answered Sep 21 '17 at 07:56

Joerg S

4,730
3
24
43

Currying groovy CPS closure for parallel execution

Solution:

3 Answers3

Linked