4

I am trying to configure a job on my local circleci (using docker executor, image: google/cloud-sdk:latest), and that job requires a sops gcp kms encrypted file to be decrypted. I have setup a google service account for the gcp kms decrypt service (I can run the script, to be run via the circleci job, successfully locally by decrypting the sops file via the service account, so I know the service account setup is valid). Here is how I am running my job.

1- I base64 encode the google service account json file: base64 path/to/service_aacount_file.json

2- I run circleci job, setting GCLOUD_SERVICE_KEY environment variable on circleci, with the base64 encoded content from the previous step: circleci local execute --env GCLOUD_SERVICE_KEY='<Base64EncodedServiceAccountJsonFileContent>' --job '<MyJob>'

3- Here is my circleci config:

- run:
          name: <MyJob>
          command: |
            apt-get install -y docker
            apt-get install -y sudo
            cd $(pwd)/path/to/jobcode
            echo $GCLOUD_SERVICE_KEY | base64 -d > ${HOME}/<MyGoogleServiceAccountJsonFile.json>
            export GOOGLE_APPLICATION_CREDENTIALS="${HOME}/<MyGoogleServiceAccountJsonFile.json>" 
            gcloud auth activate-service-account --key-file ${HOME}/<MyGoogleServiceAccountJsonFile.json>
            echo $GOOGLE_APPLICATION_CREDENTIALS
            ls -halt $GOOGLE_APPLICATION_CREDENTIALS
            cat $GOOGLE_APPLICATION_CREDENTIALS
            sudo ./<RunJob.sh>

4- I get following error when I execute the job:

Failed to get the data key required to decrypt the SOPS file.

Group 0: FAILED
  projects/<MyProject>/locations/<MyLocation>/keyRings/<MySopsKeyring>/cryptoKeys/<MyKey>: FAILED
    - | Cannot create GCP KMS service: google: could not find
      | default credentials. See
      | https://developers.google.com/accounts/docs/application-default-credentials
      | for more information.

Recovery failed because no master key was able to decrypt the file. In
order for SOPS to recover the file, at least one key has to be successful,
but none were.

5- Further, from the console output:

a- I can see that the service account was successfully activated: Activated service account credentials for: [<MyServiceAccount>@<MyProject>.iam.gserviceaccount.com]

b- The GOOGLE_APPLICATION_CREDENTIALS environment variable is set to the service account json's path: /path/to/service_account.json

c- The above file has been correctly base64 decoded and contains valid json:

{
    "client_x509_cert_url": "<MyUrl>",
    "auth_uri": "<MyAuthUri>",
    "private_key": "<MyPrivateKey>",
    "client_email": "<ClientEmail>",
    "private_key_id": "<PrivateKeyId>",
    "client_id": "<ClientId>",
    "token_uri": "<TokenUri>",
    "project_id": "<ProjectId>",
    "type": "<ServiceAccount>",
    "auth_provider_x509_cert_url": "<AuthProviderCertUrl>"
}

6- Some other things I have tried:

a- Tried setting google project name in environment variables, but still same error.

b- Tried setting GOOGLE_APPLICATION_CREDENTIALS to file's content, instead of file path, but again same result.

c- Tried setting GOOGLE_APPLICATION_CREDENTIALS by providing file path without quotes or single quotes, but still no difference.

d- Tried setting $BASH_ENV by doing echo 'export GOOGLE_APPLICATION_CREDENTIALS=path/to/service_account.json' >> $BASH_ENV, but same error

Please help.

ltcolumb
  • 73
  • 1
  • 7
  • 1
    This error `Cannot create GCP KMS service: google: could not find` indicates that the required library is not present. – John Hanley Feb 09 '20 at 17:10
  • @John Hanley That's strange, I'm using the ```google/cloud-sdk:latest``` docker image, which I'd thought would contain all the gcp services. – ltcolumb Feb 10 '20 at 10:46
  • Try to copy `path/to/service_account.json` to `/root/.config/gcloud/application_default_credentials.json` and then try to run `gcloud auth application-default` – Serhii Rohoza Feb 10 '20 at 15:41

1 Answers1

2

Five options that could work:

  1. Try to run the following command: gcloud auth application-default login
  2. Try this command to set the env var: echo 'export GOOGLE_APPLICATION_CREDENTIALS=/tmp/service-account.json' >> $BASH_ENV
  3. The other thing is that I see that runjob.sh is running under root. It could be that the gcp credentials are not visible under sudo per default. Either run the script without sudo or run the preceding commands with sudo.
  4. As a last resort (those options worked for me, could be different in your scenario): { echo 1; echo 1; echo n; } | gcloud init
  5. gcloud components update This sometimes works when the sdk is outdated.
  6. config set project [PROJECT_NAME]

You can also check active accounts with: gcloud auth list

Cloudkollektiv
  • 11,852
  • 3
  • 44
  • 71
  • Thanks, but unfortunately none of these worked. Here's what I get for each. – ltcolumb Feb 10 '20 at 10:05
  • gcloud auth application-default login: Results in ```Activated service account credentials for: [@.iam.gserviceaccount.com] The environment variable [GOOGLE_APPLICATION_CREDENTIALS] is set to: [path/to/service_account.json] Credentials will still be generated to the default location: [/root/.config/gcloud/application_default_credentials.json] To use these credentials, unset this environment variable before running your application.``` It gets stuck at this point, I had to ctrl-c to kill it. – ltcolumb Feb 10 '20 at 10:13
  • Number 2, I'd tried that already (I've updated question to reflect that), but got same error as OP. – ltcolumb Feb 10 '20 at 10:24
  • Added a fourth and fifth "last resort" options, which worked for me when using gcloud and a service account on an auto-generated vm. – Cloudkollektiv Feb 10 '20 at 12:09
  • Thanks @Nebulastic, but unfortunately still no luck and still getting same error :( – ltcolumb Feb 10 '20 at 12:52
  • Could you set the project explicitly and you could also see if any user is authenticated with gcloud auth list. – Cloudkollektiv Feb 10 '20 at 13:06
  • 1
    gcloud auth list shows me this: ```Credentialed Accounts ACTIVE ACCOUNT * @.iam.gserviceaccount.com``` – ltcolumb Feb 10 '20 at 14:10
  • 1
    I've tried setting project explicitly using ```gcloud config set project ``` but still get same error. – ltcolumb Feb 10 '20 at 14:11