Google Cloud Build + Google Secret Manager Substitution Problems

Question

we have a repository that needs to go get a private repo. To do this, we are using an SSH key to access the private repo/module.

We are storing this SSH key using Google Secret Manager and passing it to Docker using the build-arg flag. Now, when we do this locally, the Dockerfile builds and runs as intended. This is the command we use for a local build:

export SSH_PRIVATE_KEY="$(gcloud secrets versions access latest --secret=secret-data)" && \
docker build --build-arg SSH_PRIVATE_KEY -t my-image .

However, when we try to move this setup to Google Cloud Build, we run into 403 forbidden errors from Bitbucket, which leads me to believe that the SSH key is either not being read or formatted correctly.

The full 403 error is:

https://api.bitbucket.org/2.0/repositories/my-repo?fields=scm: 403 Forbidden
Step #0 - "Build":  server response: Access denied. You must have write or admin access.

What is even stranger is that when I run the Cloud Build local emulator, it works fine using this command: cloud-build-local --config=builder/cloudbuild-prod.yaml --dryrun=false .

I've tried many different formats and methods, so out of desperation I am asking the community for help. What could be the problem?

Here is our cloudbuild.yaml:

steps:
# Get secret
  - id: 'Get Secret'
    name: gcr.io/cloud-builders/gcloud
    entrypoint: 'bash'
    args:
      - '-c'
      - |
          gcloud secrets versions access latest --secret=secret-data > /workspace/SSH_PRIVATE_KEY.txt

# Build
  - id: 'Build'
    name: 'gcr.io/cloud-builders/docker'
    entrypoint: 'bash'
    args:
      - '-c'
      - |
          export SSH_PRIVATE_KEY=$(cat /workspace/SSH_PRIVATE_KEY.txt) &&
          docker build --build-arg SSH_PRIVATE_KEY -t my-image .

I’d recommend you set bash flags -e, -E, and -o pipefail. Did you give Cloud Build permissions to access the secret? — sethvargo, Sep 09 '20 at 23:16
@MrTech, I updated the question to include the errors. If you want a specific build ID, let me know how to get in touch. — Mike, Sep 09 '20 at 23:54
@sethvargo, thanks - yes, cloud build has permissions and is grabbing the secret just fine. — Mike, Sep 09 '20 at 23:57
Can you try dropping the &&? Also I *think* build-arg might ned to be SSH_PRIVATE_KEY=$SSH_PRIVATE_KEY — sethvargo, Sep 10 '20 at 00:52
Thanks @sethvargo tried both of those suggestions but still have the same error — Mike, Sep 10 '20 at 00:57

score 0 · Answer 1 · answered Sep 10 '20 at 08:06

With Cloud Build, when you want to get local linux variable, and not the substitution variable, you have to espace the $ with another $. Look at this:

# Build
  - id: 'Build'
    name: 'gcr.io/cloud-builders/docker'
    entrypoint: 'bash'
    args:
      - '-c'
      - |
          export SSH_PRIVATE_KEY=$(cat /workspace/SSH_PRIVATE_KEY.txt) 
          docker build --build-arg $$SSH_PRIVATE_KEY -t my-image .

The SSH_PRIVATE_KEY is prefixed by $$ to say: don't look at the substitution variable, but look at the linux variable.

I also remove the && at the end of the export line. The pipe | means: Run each command in succession, line return limit each command

score 0 · Accepted Answer · answered Sep 10 '20 at 21:37

Thanks for all the help! This one was pretty weird. Turns out it's not an issue with Cloud Build or Secret Manager but the Dockerfile I was using.

Instead of setting GOPRIVATE with the command in the Dockerfile below, I was using a statement like RUN export GOPRIVATE="bitbucket.org/odds".

In case anyone runs into something like this again, here's the full Dockerfile that works.

FROM golang:1.15.1

WORKDIR $GOPATH/src/bitbucket.org/gml/my-srv

ENTRYPOINT ["./my-srv"]

ARG CREDENTIALS

RUN git config \
    --system \
    url."https://${CREDENTIALS}@bitbucket.org/".insteadOf \
    "https://bitbucket.org/"

RUN go env -w GOPRIVATE="bitbucket.org/my-team"

COPY . .

RUN make build

Google Cloud Build + Google Secret Manager Substitution Problems

2 Answers2