6

I am using GKE(Google Kubernetes Engine) 1.13.6-gke.6 and I need to provide etcd encryption evidence for PCI purposes.
I have used --data-encryption-key flag and used a KMS key to encrypt secrets following this documentation.

I need to give a set of commands which will prove that the information stored in etcd of the master node is encrypted.

Here is how we verify that the secrets stored inside a normal Kuebrnetes Cluster (not GKE) are encrypted.

As we know GKE is a managed service and master node is managed by GCP. Is there a way to access GKE "etcd" to see the stored secrets and data at rest ?

Amit Yadav
  • 4,422
  • 5
  • 34
  • 79
  • You can create a K8s cluster with a master and few worker nodes in (GCP) your account. Master node will install `etcd` as its part of master component. Then you create some secrets, configMaps which will be stored in `etcd`. Then you may play with `etcd` to find out how they are being stored. +1 good question. – Robert Ranjan Jun 20 '19 at 02:51
  • @RobertRanjan but we will not be using GKE this way, right? The master node is independent of Google Cloud and this way I would not be able to verify if the data is encrypted inside GKE. – Amit Yadav Jun 20 '19 at 05:07
  • I never used GKE. If we don’t have access to GKE’s etcd, we can’t verify using the way I mentioned above. – Robert Ranjan Jun 20 '19 at 05:38
  • That is what I am looking for. A way to access the etcd or to verify the encryption by any other means. – Amit Yadav Jun 20 '19 at 05:42
  • If GKE is a service, it's `etcd` holds information about many more cluster just like yours. So, NO ONE should be having access to it except their admins. GKE should be having their own PCI compliance and much more to protect customer/cluster data. "You should not try to get access to GKE's etcd". You should look for other ways like reading GKE's documentation or reaching out to them to find out what you are looking for. – Robert Ranjan Jun 20 '19 at 06:43
  • @RobertRanjan But I am just looking for a confirmation on the encryption part or that there is no way to access GKE etcd directly. That should be enough for this task. However, your points make sense. – Amit Yadav Jun 20 '19 at 07:42

2 Answers2

6

Why do you have to prove that the information is encrypted? GKE is covered by Google Cloud's PCI DSS certification and since the master is a part of the "cluster as a service" that should be out of scope for what you need to show since you don't (and can't) control the way in which the storage is implemented.

One thing you can do is use Application-layer Secrets Encryption to encrypt your secrets with your own key stored in Cloud KMS. For your secrets you would be able to run commands to prove that additional level of encryption.

Robert Bailey
  • 17,866
  • 3
  • 50
  • 58
  • Thank you for the PCI link. It mostly does the work for me. However, I am just curious to learn. I followed this (https://cloud.google.com/kubernetes-engine/docs/how-to/encrypting-secrets) documentation and created a cluster. Create a kuberntes secret using `kubectl create secret` command, is there a way in which I can verify the secret stored inside the k8 cluster is encrypted? – Amit Yadav Jun 19 '19 at 14:10
  • 1
    Yeah, it looks like you are right and that the encryption / decryption happens behind the scenes. The one thing you can do is destroy your key to prove to yourself that it is working, but that will render the encrypted data unusable so you can't do that on your production clusters. – Robert Bailey Jun 19 '19 at 16:35
  • 1
    Even after destroying the key in KMS, I am able to access the secrets `kubectl get secrets -o yaml`. I am still not able to prove anything. – Amit Yadav Jun 20 '19 at 05:44
  • The how-to you linked to says "When you destroy a key in Cloud KMS used to encrypt a Secret in GKE, that Secret is no longer available" so if you can show that isn't happening please file a bug with Google's Cloud support team. – Robert Bailey Jun 20 '19 at 21:07
  • `WARNING: Cluster etcd-cluster is DEGRADED with message: "Codes: [None] Messages: [u'Kubernetes Engine Service Agent account unable to use CloudKMS key configured for Application Level encryption. Please, refer to https://cloud.google.com/kubernetes-engine/docs/how-to/encrypting-sec rets to fix this.'].".` This is the error message I see when I do `gcloud container cluster list` but every `kubectl command` works. I think I will report a bug or a query for sure. – Amit Yadav Jun 21 '19 at 05:55
0

Ref: https://cloud.google.com/kubernetes-engine/docs/concepts/control-plane-security#etcd_security

In Google Cloud, customer content is encrypted at the filesystem layer by default. So disks that host etcd storage for GKE clusters are encrypted at the filesystem layer. For more information, see Encryption at Rest.

Karthik Murugan
  • 1
  • 1