How to run parallel stages in isolated pods for a declarative jenkins pipeline

Question

I'm trying to run some end-to-end tests in parallel and on DIFFERENT kubernetes pods on a declarative jenkins pipeline, jenkins however seems to attempt running the parallel stages on the SAME kubernetes pod. This leads to database deadlocks since both processes try inserting/truncating/updating/querying the same tables. Is there a way I can spin up a different pod for each of the parallel stages?

The kubernetes-plugin config:

  agent {
  kubernetes {
    label 'my-label'
    defaultContainer 'jnlp'
    yaml """
apiVersion: v1
kind: Pod
metadata:
  name: dind
spec:
  containers:
    - name: < default container >
      image: < image >
      securityContext:
          privileged: true
          fsGroup: 1000
      command:
      - cat
      tty: true
      volumeMounts:
        - name: jenkins-bundle-gems
          mountPath: /usr/local/bundle


    - name: <tests-container-name>
      image: < image > 
      securityContext:
          privileged: true
          fsGroup: 1000
      volumeMounts:
        - name: jenkins-bundle-gems
          mountPath: /usr/local/bundle
      command:
      - cat
      tty: true
"""
 }
   }

The parallel stage:

      stage('Test'){
        parallel {
          stage("Branch 1") {
              steps {
                container('<tests-container-name>') {
                  sh "jenkins/scripts/initdb.sh"
                  sh 'bundle exec rspec --exclude-pattern "spec/features/*_spec.rb" spec'
                }
              }
            }
            stage("Branch 2") {
              steps {
                container('<tests-container-name>') {
                  sh "jenkins/scripts/initdb.sh"
                  sh "bundle exec rspec `jenkins/scripts/split_features.sh 0`"
                }
              }
            }
        }
      }

EXPECTATION: I would like for jenkins to spin up two different pods for each of the parallel stages. This would allow me to use different databases for each of the tests.

ACTUAL RESULT: Jenkins runs both stages concurrently on the same pod.

Hi Marc, you defined only one pod with 2 containers. According to your stage you are using that same container. I believe you should check the option for side car (or 2 side cars) also known as init containers to fullfill your needs — Chen, Mar 27 '19 at 17:17
@Chen What if I defined 5 containers ( from the same image) on the same pod. Would that solve the concurrency issue? — martinkaburu, Mar 29 '19 at 10:08
I think you are missing the point. I suggest you should change the yaml file and the jenkinsfile. On the Jenkinsfile use different containers instead the same onr — Chen, Apr 01 '19 at 14:38

score 3 · Answer 1 · answered Sep 14 '19 at 16:28

You have__________: stage > parallel > stage > steps.

You need to have also: stage > parallel > stage > agent.

Do not repeat pod definition twice, it is recommended to put the pod definition in a separated file and refer to it using yamlFile instead of yaml :

  stage('Test'){
    parallel {
      stage("Branch 1") {
          agent {
             kubernetes {
               defaultContainer 'jnlp'
               yamlFile 'Jenkins.pod.yaml'
             }
          }
          steps {
            container('<tests-container-name>') {
              sh "jenkins/scripts/initdb.sh"
              sh 'bundle exec rspec --exclude-pattern "spec/features/*_spec.rb" spec'
            }
          }
        }
        stage("Branch 2") {
          agent {
             kubernetes {
               defaultContainer 'jnlp'
               yamlFile 'jenkins.pod.yaml'
             }
          }
          steps {
            container('<tests-container-name>') {
              sh "jenkins/scripts/initdb.sh"
              sh "bundle exec rspec `jenkins/scripts/split_features.sh 0`"
            }
          }
        }
    }
  }

Hint

if blueocean is one of your plugins, it will help you to draw your pipeline under http://HOST/blue/organizations/jenkins/pipeline-editor/, then you can copy the Jenkinsfile code by typing [Cmd + s]

score 2 · Answer 2 · answered Mar 27 '19 at 15:40

Try something like this

stage("Run additional parallel tests") {
        parallel( 
            "parallel stage 1": {
                [INSERT YOUR CODE HERE]
            },

            "parallel stage 2": {
                [INSERT YOUR CODE HERE]
            }
        )
    }
}

score 1 · Answer 3 · answered Jul 17 '19 at 16:11

1

You can set agent {} for each parallel stage to launch a pod per stage.

answered Jul 17 '19 at 16:11

Morgan Christiansson

29,280
1
19
13

please give a boilerplate like [George Answer](https://stackoverflow.com/a/55381198/747579) I have a [gist here](https://gist.github.com/abdennour/4a1c2ac93fbe7edc36768a5ae80a2671) to present all ways of parallelism – Abdennour TOUMI Sep 14 '19 at 13:24

How to run parallel stages in isolated pods for a declarative jenkins pipeline

3 Answers3