Passing a private key through Fly Cli

Question

The company I work for has recently started to use Concourse CI to do all our CI needs. At the current moment one of my jobs consists of a task with a script that scp's and ssh's into our aws ec2 instances and configures those servers. The issue I am having, however, is getting the private key to ssh into those instances. One way discussed here (https://concourse-ci.org/fly-set-pipeline.html) is to pass the key in through a variable. In my script, I take that variable passed in and echo it to a new .pem file and I set the permissions to 600. When I echo just the variable and later cat the new .pem file, they look exactly the same as the original .pem file. The container I am trying to ssh from is the standard ubuntu docker image.

When I try to use this file to scp and ssh I am confronted with the prompt about entering the passphrase. If I try to ssh with the original file I don't get this prompt at all. Is there something I am missing? I would greatly appreciate some insight into this issue.

pipeline.yml

jobs:
- name: edge-priceconfig-deploy
  plan:
  - aggregate:
    - get: ci-git
    - get: pricing-config
  trigger: true
  - task: full-price-deploy
    file: ci-git/ci/edge/edge-price-config-task.yml
    params:
      USER_AND_SERVER: {{edge_user_and_server}}
      DEPLOY_KEY_PAIR: {{deploy_key_pair}}

task.yml

---
platform: linux

image_resource:  
  type: docker-image
  source: {repository: ubuntu}

inputs:
- name: ci-git
- name: pricing-config

run:
  path: ./ci-git/ci/edge/edge-priceconfig-deploy.sh

task.sh

#!/bin/bash
touch DeployKeyPair.pem

echo $DEPLOY_KEY_PAIR
echo $DEPLOY_KEY_PAIR > DeployKeyPair.pem
cat DeployKeyPair.pem

apt-get update && apt-get -y install sudo
sudo apt-get -y install openssh-client
sudo chmod 400 ci-git/key/DeployKeyPair.pem
sudo chmod 600 DeployKeyPair.pem

mkdir company-price-config-edge
mv pricing-config/fsconfig/conf/com.company.api.v1.pricing/*.xlsx company-price-config-edge/

commandstr="sudo rm -f /etc/company/edge/fsconfig/*xlsx; \
        ls -l /etc/company/edge/fsconfig; \
        sudo mv /home/ec2-user/company-price-config-edge/*xlsx /etc/company/edge/fsconfig/; \
        sudo rm -rf /home/ec2-user/company-price-config-edge;"


scp_link="$USER_AND_SERVER:/home/ec2-user/"
scp_link="$(echo $scp_link | tr -d ' ')"
echo $scp_link

sudo echo -ne '\n' | scp -r -oStrictHostKeyChecking=no -i DeployKeyPair.pem company-price-config-edge $scp_link
sudo ssh -oStrictHostKeyChecking=no -i DeployKeyPair.pem $USER_AND_SERVER $commandstr

credentials.yml

username: |
  username

password: |
  password

access_token: |
  token

ci_scripts_github: |
  ci-script-link

edge_user_and_server: |
  server.com

staging_user_and_server: |
  staging

training_user_and_server: |
  training

production_user_and_server: |
  production

deploy_key_pair:
  -----BEGIN RSA PRIVATE KEY-----
  ...
  -----END RSA PRIVATE KEY-----

Can we see your `pipeline.yml`, `task.yml`, and (sanitized) vars file? — materialdesigner, May 17 '17 at 16:55
@ChristopherHendrix I edited the original post to contain those files — Jdurham2843, May 17 '17 at 17:14

score 1 · Answer 1 · answered May 17 '17 at 17:30

1

Two things.

One: You need to declare that these particular environment variables are being used by your task

task.yml:

---
platform: linux

image_resource:  
  type: docker-image
  source: {repository: ubuntu}

inputs:
- name: ci-git
- name: pricing-config

params:
  USER_AND_SERVER: 
  DEPLOY_KEY_PAIR:

run:
  path: ./ci-git/ci/edge/edge-priceconfig-deploy.sh

Two: I think you need to add a | to your credentials.yml file.

credentials.yml:

deploy_key_pair: |
  -----BEGIN RSA PRIVATE KEY-----
  ...
  -----END RSA PRIVATE KEY-----

answered May 17 '17 at 17:30

materialdesigner

1,492
10
13

I added the changes you made and still no luck. I have the key sitting in a git repository and can use that one just fine. Even when I make a copy of that file and use it as the key it works without a hitch. However, something about echoing the key from the variable into a file causes us issues. – Jdurham2843 May 17 '17 at 18:34
@ChristoperHendrix Sorry to get back so late! I tried the echo -n and still had no luck. When echoing over the contents from one private key file to another, might there be something going on behind the scenes, like some kind of metadata being written, that might be keeping the the aws instance from recognizing the new pem file as the original? – Jdurham2843 May 19 '17 at 05:25

score 1 · Accepted Answer · answered Jun 02 '17 at 20:31

Had this same problem today and took me a bit to figure out what was going on.

You're specifying the private key as a multiline yaml property, but when it's being echoed out to a file it's stripping out the linebreaks and replacing them with spaces, so your key is a large single line file.

I had to use sed to replace the space with newlines to get it to work

echo $DEPLOY_KEY_PAIR | sed -e 's/$KEY-----$\s/\1\n/g; s/\s$-----END$/\n\1/g' | sed -e '2s/\s\+/\n/g' > DeployKeyPair.pem

First sed command uses two sets of replacement groups, both with the same purpose

First group - Grab header text and break it out to a new line

s/$KEY-----$\s/\1\n/g

$KEY-----$ puts the last part of the header block in a capture group
\s matches whitespace
/1\n/g Puts capture group (#1) back in, then adds a newline
/g Unnecessary global capture that I forgot to remove

Second sed group does the same but grabs the space and first part of the footer text to put it on a new line

Third group does a global replacement on whitespace and replaces it with newline. This had to be a separate command so processing was finished for first command and it was then a 3-line file/stream

'2s/\s\+/\n/g'

2s/ Perform replacement only on line 2
\s\+ matches any whitespace
\n replace with newline
/g global match to grab every instance

Thanks, I tried your solution and it worked like a charm! When I was outputting the Deploy Key earlier, it looked like it had the same formatting at the original pem file, but I guess the window size just happened to be big enough to cause wrap around, making the key appear to be the same as the original. — Jdurham2843, Jun 16 '17 at 18:24

score 0 · Answer 3 · answered May 04 '18 at 09:57

0

you can also use a keystore integration, e.g. credhub, with concourse.

see https://docs.pivotal.io/p-concourse/credential-management.html

answered May 04 '18 at 09:57

Dirk Hoffmann

1,444
17
35

Passing a private key through Fly Cli

3 Answers3