Jenkins - Multibranch Pipeline unexplained failure

Question

In my pipeline, I make use of readJSON and writeJSON methods of the Pipeline Utility Step plugin to read/write files in pipeline stages. For a normal pipeline everything works great. However, when I create a "multibranch pipeline", the same pipeline fails. The failure happens when a subsequent stage attempts to read a file, using readJSON, that should have been produced by a previous stage. The curious part is: within one stage, a file that was produced was available to be used by a command (the command aws ecs register-task-definition succeeds) BUT is not present in the workspace /var/jenkins_home/workspace/TestPipeline on the node afterwards.

To focus on the failure, I am only pasting the relevant stages of the pipeline.

stage('RegisterTaskDefinition') {
    agent any
    steps {
        sh 'printenv'
        script {            
            def templateFile = env.TEMPLATE_BASE_PATH +'/' + TASK_DEF_TEMPLATE
            def taskDefinitionTemplate = readJSON(file: templateFile)

            taskDefinitionTemplate.taskRoleArn = env.TASK_ROLE_ARN
            taskDefinitionTemplate.executionRoleArn = env.EXECUTION_ROLE_ARN
            taskDefinitionTemplate.containerDefinitions[0].image = "NewImage"
            taskDefinitionTemplate.containerDefinitions[0].portMappings[0].containerPort = env.APP_PORT.toInteger()

            taskDefFile = env.TEMPLATE_BASE_PATH + '/' + env.TASK_DEFINITION_FILE
            writeJSON(file: taskDefFile, json: taskDefinitionTemplate)

            def registerTaskDefinitionOutput = sh (
            script: "aws ecs register-task-definition --cli-input-json file://${taskDefFile}",
            returnStdout: true
            ).trim()
            echo "Register Task Def result: ${registerTaskDefinitionOutput}"

            def registerTaskDefOutputFile = env.TEMPLATE_BASE_PATH + '/registerTaskDefOutput.json'
            echo "********************************"
            sh 'pwd'
            writeJSON(file: registerTaskDefOutputFile, json: registerTaskDefinitionOutput, pretty: 2)
            echo "********************************${registerTaskDefOutputFile}"
        }
    }
}
stage('CreateTaskSet') {
    agent any
    steps{
        script{
            def registerTaskDefOutputFile = env.TEMPLATE_BASE_PATH + '/' + env.REGISTER_TASK_DEF_OUTPUT
            def taskSetTemplateFile = env.TEMPLATE_BASE_PATH + '/' + env.TASK_SET_TEMPLATE_FILE
            def taskSetFile = env.TEMPLATE_BASE_PATH + '/' + env.TASK_SET_FILE

            def registerTaskDefinitionOutput = readJSON(file: registerTaskDefOutputFile)
            def taskSetTemplateJson = readJSON(file: taskSetTemplateFile)

            taskSetTemplateJson.taskDefinition = registerTaskDefinitionOutput.taskDefinition.taskDefinitionArn
            taskSetTemplateJson.loadBalancers[0].containerPort = env.APP_PORT.toInteger()
            taskSetTemplateJson.loadBalancers[0].targetGroupArn = targetGroupArn

            writeJSON(file: taskSetFile, json: taskSetTemplateJson, pretty: 2)
        }
    }
}

To clarify, the RegisterTaskDefinition stage succeeds and the failure happens for CreateTaskSet stage.

The exception generated is:

java.io.FileNotFoundException: /var/jenkins_home/workspace/TestPipeline/infrastructure/registerTaskDefOutput.json does not exist.
    at org.jenkinsci.plugins.pipeline.utility.steps.json.ReadJSONStepExecution.doRun(ReadJSONStepExecution.java:82)
    at org.jenkinsci.plugins.pipeline.utility.steps.AbstractFileOrTextStepExecution.run(AbstractFileOrTextStepExecution.java:32)
    at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

I have the following questions:

1. What explains the behavioral difference b/w the two pipeline types?
2. For a normal pipeline, I can see the files produced by the pipeline under /var/jenkins_home/workspace/TestPipeline on the node. For a "multibranch pipeline" that is not the case. What directory is Jenkins using for a "multibranch pipeline"?
3. What logs would help me diagnose the issue?

I am unable to find any documentation that explain this difference in behavior b/w the two pipeline types. I would appreciate any suggestions.

Answer 1

From your code:

stage('RegisterTaskDefinition') {
    agent any

and

stage('CreateTaskSet') {
    agent any

This is entirely possible that these two stages run on different agents, since you have told Jenkins that you want a random agent to run the first stage and a random agent to run a second stage. These agents might have been the same previously; however there's no requirement they will be the same.

To prevent this, allocate a node early and don't change it:

pipeline {
   agent any
   stage('1') { ... }
   stage('2') { ... }
}

This guarantees that both stages will run on the same agent, so the results that the first stage writes in ${env.WORKSPACE} will be available to the second one.