ETCD restore granularity

Question

I'm studying some backup and restore solutions for Kubernetes and Openshift, typically with Velero. So, I'm not familiar with ETCD backups, but I would like to ask about the granularity level of etcd snapshots restore. Does ETCD snapshots only allow to restore the cluster state as a whole? or we can restore some specific objects like namespaces, deployments, pod, pvc, etc? In another way, can ETCD backups do the same job as velero (except what concerns volumes) or not ?

Answer 1

Etcd snapshots and restore are not meant for partial recovery, it's all or nothing.

However you may use an etcd snapshot to retrieve those data you've lost and re-create them in your cluster, without resetting etcd.

One way to do so would be to restore your etcd database into a container, on your workstation. Start a container, eg:

docker -v $(pwd)/restore-path:/restore run --entrypoint /bin/sh -it quay.io/coreos/etcd:vX.Y.Z 

Restore your snapshot, start etcd, then query it:

etcdctl snapshot restore /restore/snapshot.db 
etcd --name default --listen-client-urls http://localhost:2379  --advertise-client-urls http://localhost:2379 --listen-peer-urls http://localhost:2380 &
etcdctl get / --prefix --keys-only

You should get back the list of objects, as stored in etcd. Now try to locate the objects you want to restore:

etcdctl get / --prefix --keys-only | grep my-object-name

etcdctl get /kubernetes.io/configmaps/my-namespace/lost-config  --print-value-only | tee -a /restore/my-lost-config.out

This output is a bit messy, as it would return some non-printable characters alongside your YAML. Still it should be good enough recovering a few objects. Otherwise you can look into something like Augher (https://github.com/jpbetz/auger), which should clean this up for you, if you need something robust scripting such recoveries, ...

Answer 2

If you want to take etcd backup in JSON format that you can easily restore to any cluster, you can use this simple backup and restore utility that I have developed: simple-etcd-backup-restore

This utility makes the backup and restore process easier and faster. You can also compress/decompress the backup file and upload/download it on/from AWS S3 bucket. The size of backup files also gets smaller. Data in the backup files stays readable as well.