0

I followed this document to create a GKE cluster (1.13.6-gke.6) with --database-encryption-key flag giving a KMS key for enabling Application-layer Secrets Encryption.

I created a secret using the following command:

kubectl create secret generic dev-db-secret --from-literal=username=someuser --from-literal=password=somepass

So if my assumption is correct, these secrets are stored encrypted using the KMS key provided by me while creating the cluster. However, even after I have destroyed all the versions of the used key, I am able to see the secret stored inside the GKE etcd using kubectl get secret dev-db-secret -o yaml as well as I am able to see them in a pod created using the below manifest:

apiVersion: v1
kind: Pod
metadata:
  name: secret-env-pod
spec:
  containers:
  - name: mycontainer
    image: redis
    env:
      - name: SECRET_USERNAME
        valueFrom:
          secretKeyRef:
            name: dev-db-secret
            key: username
      - name: SECRET_PASSWORD
        valueFrom:
          secretKeyRef:
            name: test-secret
            key: password
  restartPolicy: Never

If I exec into the above pod and do echo SECRET_USERNAME and echo SECRET_PASSWORD I get the username and password printed on my console in plain text.

Is this the way the encryption supposed to work? If yes, where is the encryption happening exactly? What am I doing wrong? Are the secrets really encrypted?

Community
  • 1
  • 1
Amit Yadav
  • 4,422
  • 5
  • 34
  • 79

1 Answers1

1

I'm not 100% sure, but I think those keys are cached so it's probably will take a while before the decryption will fail. This is the case for Azure, I guess it's similar for GKE.

BTW you might want to read how to protect the manifest files so you can store them on Git. I wrote a blog post describing some of the options you can use.

Omer Levi Hevroni
  • 1,935
  • 1
  • 15
  • 33
  • It has been a long time (more than 12-13 hours) and decryption doesn't fail. However, I am not able to create a new secret but can `get`,`delete` and `modify` it. Also, new pods created can use the secret even after the Cluster gives warning that the key is not available. – Amit Yadav Jun 21 '19 at 07:09
  • I would file an issue under the relevant [KMS plugin](https://github.com/GoogleCloudPlatform/k8s-cloudkms-plugin). Not sure if this is the right plugin, but this is the first thing I find on Google. Or contacting GKE support. Please add an answer when you find out what happened :) – Omer Levi Hevroni Jun 23 '19 at 05:46