Google KMS on AppEngine/Python & Development AppServer

Question

It's not clear from the documentation how one might wield Google Key Management System (KMS) on Google App Engine Standard, particularly when developing locally using the development server.

It would appear as reasonably straightforward as:

1. Installing google-api-python-client in a Python virtual env (and adding the virtualenv path with google.appengine.ext.vendor in appengine_config.py)
2. importing googleapiclient.discovery
3. getting the application identity with google.appengine.api.app_identity
4. Using the kms client in the anticipated / documented way

... then following the tutorial linked in the Documentation. However my attempts so far have not resulted in success, and it appears the documentation is wanting for a few steps.

It feels like I'm breaking new ground that I'm sure others must have already.

Has anyone documented using Google KMS on App Engine Standard & its local development server?

EDIT - Update with Code Example

Here's some code that illuminates -- the problem would appear to be with my setup of default credentials.

mykms.py

import googleapiclient.discovery
from google.appengine.api import app_identity

from oauth2client.client import GoogleCredentials
credentials = GoogleCredentials.get_application_default()

PROJECT = 'my-crypto-project'
IS_LOCAL = True
LOCATION = 'global'
TESTING_KR = 'testing-keyring'
KEY_RING = TESTING_KR if IS_LOCAL else app_identity.get_application_id()

kms = googleapiclient.discovery.build('cloudkms', 'v1', credentials=credentials)

def encrypt(plaintext, cryptokey, keyring=KEY_RING, location=LOCATION):
    name = 'projects/{}/locations/{}/keyRings/{}/cryptoKeys/{}'.format(
        PROJECT, location, keyring, cryptokey
    )
    cryptokeys = kms.projects().locations().keyRings().cryptoKeys()
    request = cryptokeys.encrypt(name=name, body={'plaintext': plaintext})
    return request.execute()


def decrypt(ciphertext, cryptokey, keyring=KEY_RING, location=LOCATION):
    name = 'projects/{}/locations/{}/keyRings/{}/cryptokey'.format(
        PROJECT, location, keyring
    )
    cryptokeys = kms.projects().locations().keyRings().cryptoKeys()
    request = cryptokeys.decrypt(name=name, body={'ciphertext': ciphertext})
    return request.execute()

Now calling, via dev_appserver.py:

import mykms
mykms.encrypt("my text", cryptokey="my-key-ring")

gives an error of:

HttpError: https://cloudkms.googleapis.com/v1/projects/np-crypto/locations/global/keyRings/localhost-testing/cryptoKeys/machine-identifiers:encrypt?alt=json returned "Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.">

That's not especially helpful, being mostly concerned with Google Sign-In on the website; however, when I import the mykms from the command line, I get the error:

The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

This seems like the correct lead for now. Will flush it out and report back.

EDIT #2

The application seems to now connect to KMS. I deleted and re-logged into gcloud auth application-default login.

However, there's a weird side effect — something seems to be scanning the drive, and hundreds of messages (seemingly one for every accessible directory from root) like the following clutter the log:

INFO 30 Jun 2017 20:06:57 Sandbox prevented access to file "/Users"

INFO 30 Jun 2017 20:06:57 If it is a static file, check that application_readable: true is set in your app.yaml

Answer 1

If you're developing using Cloud KMS in GAE, there isn't a local dev service, you can only talk to the main production service as you've gathered. You could use the libraries as you've detailed to develop locally, but would still be hitting production.

Note that you'll have to give GAE application default credentials with a scope for use, see https://cloud.google.com/kms/docs/accessing-the-api#google_app_engine

You can also make requests as the GAE service account if you use gcloud iam service-accounts keys and gcloud auth activate-service-account.

In general, for a dev environment, you might want to segment this as a separate KeyRing (or even a separate project) from your production resources.