Does etcd's storage footprint grow linearly with respect to keys and values?

Question

I noticed that, when running some stress tests on a Kubernetes cluster, etcd snapshot sizes didnt increase much, even as I added more and more stuff to my cluster.

I collected snapshots via:

etcdctl --endpoints="https://localhost:2379" --cacert="/etc/kubernetes/pki/etcd/ca.crt" -cert="/etc/kubernetes/pki/etcd/server.crt" --key=/etc/kubernetes/pki/etcd/server.key snapshot save jay.db

And compared them:

root@tkg-mgmt-vsphere-20221014024846-control-plane-mp642:/home/capv# ls -altr jay*
-rw------- 1 root root 34975776 Oct 24 17:33 jay.db
-rw------- 1 root root 35061792 Oct 24 17:55 jay2.db
-rw------- 1 root root 35217440 Oct 24 18:05 jay3.db

So... since im putting large amounts of data into my cluster in these tests... i was wondering, does etcd storage usage grow linearly ? Or is it somehow compressed over time such that it doesnt ever "Get that big".

Ive seen related questions, such as etcd 3.5 db_size growing constantly, where it appears that compaction keeps the size low, so I supposed my real question is....

• What are the boundaries and limits of how much work compaction can do in an ever increasing kubernetes cluster of say, 100s, 1000s, 10s of thousands of objects, and beyond?
• Does compaction do an ever better job over time, due to increased amounts of similar or duplicate information content ?

Answer 1

Compaction in etcd primarily prunes the "revisions": new values for a given key and deleted ("tombstoned") keys. This is either triggered periodically through the API Server in K8s, or if auto-compaction was configured on etcd itself.

So... since im putting large amounts of data into my cluster in these tests... i was wondering, does etcd storage usage grow linearly ? Or is it somehow compressed over time such that it doesnt ever "Get that big".

The size on disk grows somewhat linearly yes. The snapshots are just a proto blob of the in-memory representation, so they're not compressed with any algorithm.

The code is fairly centralized if you fancy to read it yourself.

etcd snapshot sizes didnt increase much, even as I added more and more stuff to my cluster.

Depends on what "stuff" you've added, was it new key/values or just merely updates to existing keys? If lots of stuff was deleted and updated, then compaction will keep the size level.

What are the boundaries and limits of how much work compaction can do in an ever increasing kubernetes cluster of say, 100s, 1000s, 10s of thousands of objects, and beyond?

That depends on many things, especially on the hardware you're running on. There are some benchmarks available here. Since you seem to be at RedHat, I can also get you in touch with our OpenShift perf&scale team, they also run such benchmarks across clouds and different versions and publish their results on "the source"^TM.