Jenkins unable to create directory '/var/lib/jenkins/.ssh' when ssh into EC2

Question

Problem: During my Jenkins process I am able to establish a connection with the EC2 instance I want to copy files to but I keep getting the following errors:

Could not create directory '/var/lib/jenkins/.ssh'

Failed to add the host to the list of known hosts (/var/lib/jenkins/.ssh/known_hosts).

and

Host key verification failed.

Background: My Jenkins job is triggered by a github webhook after I push code to 'master' branch. Jenkins reads the repo's Jenkinsfile and creates a Docker agent to build the app and then deploy the built files to an EC2 container. During the deploy phase I use Jenkin's sshagent to establish a connection and then use commands to delete the old files and then copy the new files to the EC2.

pipeline {
  agent {
    docker {
      image 'node:buster'
      args '-p 20001-20100:3000'
      args '-v /etc/passwd:/etc/passwd'
    }
  }
   environment {
    CI = 'true'
    HOME = '.'
    npm_config_cache = 'npm-cache'
  }
  stages {
    stage('Install') {
      ...install code... <<<<<<<[works, no issues]   
    stage('Build') {
      ...build code... <<<<<<<[works, no issues]   
    }
    stage('Deploy') {
      parallel {
        stage('Deploy frontend') {
        ...deploy frontend code to S3 bucket... <<<<<<<[works, no issues]   
        }

        stage('Deploy backend') {
          steps {
            dir('backend') {
               sshagent(['code_commit_key']) {
                 sh 'ssh -o StrictHostKeyChecking=no ec2-user@ecx-xx-xx-x-xx.compute-1.amazonaws.com "whoami"' <<<<<[this return ec2-user after list of errors]
                 sh 'ssh -o StrictHostKeyChecking=no ec2-user@ecx-xx-xx-x-xx.compute-1.amazonaws.com "sudo su -; pm2 delete -s order-form-nestjs; rm -rf ./dist"' <<<<<[this returns list of errors]
                 sh 'scp -r ./dist/* ec2-user@ecx-xx-xx-x-xx.compute-1.amazonaws.com:/home/ec2-user' [this returns list of errors]
                 sh 'ssh -o StrictHostKeyChecking=no  ec2-user@ecx-xx-xx-x-xx.compute-1.amazonaws.com "sudo su -; pm2 start dist/main.js --name=backend-app-nestjs"' <<<<<[this returns list of errors]
                 echo 'Ssh successful'
               }
            }
          }
        }
      }
    }
  }
}```

`less /etc/passwd | grep jenkins` check is there any jenkins user exist? — Adiii, Sep 12 '19 at 02:33
On BOTH servers it returns: [ec2-user@ip-xxx-xx-xx-xx ~]$ less /etc/passwd | grep jenkins jenkins:x:1001:1001::/var/lib/jenkins:/bin/bash. (I have x'd out the ip) — cbilliau, Sep 12 '19 at 16:21

score 1 · Accepted Answer · edited Sep 19 '19 at 14:19

Found my answer here:

By default, when the user is not specified, docker lauches [sic] the container with the user defined in the dockerfile which if not specified is root.

I added args '-u root:root -v /var/lib/jenkins/workspace/myworkspace:/tmp/' + ' -v /var/lib/jenkins/.ssh:/root/.ssh' to my docker agent code, and viola, success!:

agent {
    docker {
      image 'node:buster'
      args '-p 20001-20100:3000'
      args '-v /etc/passwd:/etc/passwd -v /etc/group:/etc/group'
      args '-u root:root -v /var/lib/jenkins/workspace/myworkspace:/tmp/' + ' -v /var/lib/jenkins/.ssh:/root/.ssh'
    }
  }

score 0 · Answer 2 · answered Sep 11 '19 at 21:01

0

Can you ssh your EC2s from your jenkins server? Can you check

answered Sep 11 '19 at 21:01

MohamedSaeed

455
1
8
11

I can now. I copied the rsa key from my Jenkins server (server a) to the server I want to deploy to (server b). Not sure if that is best practice but it worked. (What is best practice for setting this connection up?) – cbilliau Sep 12 '19 at 16:18
Even though I can ssh in I am still getting same error when running the Jenkins build. – cbilliau Sep 12 '19 at 16:39
I believe that you copied the public key from [server a] to authorized file in [server b], didn't you? Can you post those errors? – MohamedSaeed Sep 14 '19 at 00:05
I did not get any errors when connecting server A to server B. A friend did tell me that the errors I am getting above are from server A (the server Jenkins is on) and that Jenkins does not have permission to that path but when I checked my jenkins permissions to var/lib/jenkins were ```drwxrwxrwx 21 jenkins jenkins 8192 Sep 16 15:35 jenkins``` – cbilliau Sep 16 '19 at 15:45

Jenkins unable to create directory '/var/lib/jenkins/.ssh' when ssh into EC2

2 Answers2