K8s Jenkins agent pod not able to create file permission denied

Question

I am using Jenkins on K8S.

Suddenly my one of the container is failing inside the Jenkinsfile, yesterday everything was working fine, but suddently it's failing

I have not upgrade any plugins and not even install any new plugins.

Edit: I have uploaded official curl image to private repository

My jenkins file has following content

podTemplate(
    containers: [
        containerTemplate(name: 'helm', alwaysPullImage: true, image: 'private-repo.com:8023/lachlanevenson/k8s-helm:v3.4.2', command: 'cat', ttyEnabled: true),
        containerTemplate(name: 'curl', alwaysPullImage: true, image: 'private-repo.com:8023/curlimages/curl', command: 'cat', ttyEnabled: true)
    ],
    imagePullSecrets: ['registry-credentials']) {
  properties([parameters(
      [string(name: 'dockerImageTag', description: 'Docker image tag to deploy'),
       string(name: 'branchName', defaultValue: 'dev', description: 'Branch being deployed'),
       string(name: 'targetBranch', defaultValue: 'dev', description: 'Target branch against which if a PR is being raised')])])

  currentBuild.description = "branch ${params.branchName}"
  node(POD_LABEL) {
    container('helm') {
     //code  here working fine
    }
    container('curl') {
      stage('Refresh Gateway routes') {
        sh script: 'curl -X POST \\\n' +
            '  http://my-server.com/actuator/gateway/refresh \\\n' +
            '  -H \'Accept: application/json\' \\\n' +
            '  -H \'Host: my-server.com\' \\\n' +
            '  -H \'cache-control: no-cache\'', label: 'Force refresh routes cache'
      }
    }
  }
}

So my second container is just has curl command and it was working fine, but suddenly it gives me an error as below

sh: can't create /home/jenkins/agent/workspace/sling-sales/sd-config/deploy-to-qa@tmp/durable-4549c114/jenkins-log.txt: Permission denied
sh: can't create /home/jenkins/agent/workspace/sling-sales/sd-config/deploy-to-qa@tmp/durable-4549c114/jenkins-result.txt.tmp: Permission denied
mv: can't rename '/home/jenkins/agent/workspace/sling-sales/sd-config/deploy-to-qa@tmp/durable-4549c114/jenkins-result.txt.tmp': No such file or directory
touch: /home/jenkins/agent/workspace/sling-sales/sd-config/deploy-to-qa@tmp/durable-4549c114/jenkins-log.txt: Permission denied
touch: /home/jenkins/agent/workspace/sling-sales/sd-config/deploy-to-qa@tmp/durable-4549c114/jenkins-log.txt: Permission denied

When I inspect the running pod I got some details as below

Events:
  Type    Reason     Age    From                                   Message
  ----    ------     ----   ----                                   -------
  Normal  Scheduled  4m56s  default-scheduler                      Successfully assigned default/sling-sales-sd-config-deploy-to-qa-3-rksjh-mt0js-f3tk8 to sd-jenkins-cluster-node-g7ow
  Normal  Pulling    4m54s  kubelet, sd-jenkins-cluster-node-g7ow  Pulling image "private-repo.com:8023/sling/jenkins/curl"
  Normal  Pulled     4m54s  kubelet, sd-jenkins-cluster-node-g7ow  Successfully pulled image "private-repo.com:8023/sling/jenkins/curl"
  Normal  Created    4m54s  kubelet, sd-jenkins-cluster-node-g7ow  Created container curl
  Normal  Started    4m54s  kubelet, sd-jenkins-cluster-node-g7ow  Started container curl
  Normal  Pulling    4m54s  kubelet, sd-jenkins-cluster-node-g7ow  Pulling image "private-repo.com:8023/lachlanevenson/k8s-helm:v3.4.2"
  Normal  Pulled     4m54s  kubelet, sd-jenkins-cluster-node-g7ow  Successfully pulled image "private-repo.com:8023/lachlanevenson/k8s-helm:v3.4.2"
  Normal  Created    4m54s  kubelet, sd-jenkins-cluster-node-g7ow  Created container helm
  Normal  Started    4m54s  kubelet, sd-jenkins-cluster-node-g7ow  Started container helm
  Normal  Pulled     4m54s  kubelet, sd-jenkins-cluster-node-g7ow  Container image "jenkins/inbound-agent:4.3-4" already present on machine
  Normal  Created    4m54s  kubelet, sd-jenkins-cluster-node-g7ow  Created container jnlp
  Normal  Started    4m54s  kubelet, sd-jenkins-cluster-node-g7ow  Started container jnlp

Log of running jenkins pod are as below

Jan 13, 2021 2:19:57 PM hudson.remoting.jnlp.Main createEngine
INFO: Setting up agent: sling-sales-sd-config-deploy-to-qa-3-rksjh-mt0js-f3tk8
Jan 13, 2021 2:19:57 PM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Jan 13, 2021 2:19:57 PM hudson.remoting.Engine startEngine
INFO: Using Remoting version: 4.3
Jan 13, 2021 2:19:57 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using /home/jenkins/agent/remoting as a remoting work directory
Jan 13, 2021 2:19:57 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to /home/jenkins/agent/remoting
Jan 13, 2021 2:19:57 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://jenkins.sling-dev.com:8080/]
Jan 13, 2021 2:19:57 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
Jan 13, 2021 2:19:57 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting TCP connection tunneling is enabled. Skipping the TCP Agent Listener Port availability check
Jan 13, 2021 2:19:57 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
  Agent address: jenkins-master-agent
  Agent port:    50000
  Identity:      50:64:22:83:23:d1:5b:25:da:01:51:0e:7d:c8:8a:a9
Jan 13, 2021 2:19:57 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Jan 13, 2021 2:19:57 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins-master-agent:50000
Jan 13, 2021 2:19:57 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Jan 13, 2021 2:19:57 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: 50:64:22:83:23:d1:5b:25:da:01:51:0e:7d:c8:8a:a9
Jan 13, 2021 2:19:58 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected

Can someone please help me how to solve this ?

Thanks Alpesh

Even if I just kept only **single container of curl**, then also getting the same issue — Alpesh Jikadra, Jan 13 '21 at 16:46
What do you see if you add `sh 'whoami'` to the beginning of the stage for your curl container? — Shane Bishop, Jan 13 '21 at 21:49
I found the issue, Earlier I was using official **curlimages/curl** image, then I changed it to **appropriate/curl** and it's working now. Thanks — Alpesh Jikadra, Jan 14 '21 at 04:21

K8s Jenkins agent pod not able to create file permission denied

0 Answers0