Clarifying Vault key decryption process

Question

I'm trying to understand Vault workflow w.r.t. keys, e.g.: https://www.vaultproject.io/docs/concepts/seal

As I understand,

• unseal (shared) keys are provided on init
• they're used to acquire the combined key
• combined key is then used to decrypt a root (master) key (which is apparently stored in the sealed vault)
• root key is then used to decrypt the data encryption key (or a keyring which contains it?..)
• the data encryption key is then used to en/decrypt the data in Vault

I get the unseal keys on init, how can I inspect the other keys? Is it impossible / are those keys just stored somewhere internally in Vault?

Unsealing is the process of obtaining the plaintext root key necessary to read the decryption key to decrypt the data, allowing access to the Vault.

Is the data encryption key / keyring also decrypted during the unseal, or is it... maybe decrypted on each Vault operation (so only the root key is stored somewhere in plaintext after the unseal)?

Is it ok that the root key is stored in plaintext after the unseal? Or is it still protected by passwords/tokens?.. Or if it's just transiently used to decrypt the data encryption key / keyring, then how are those protected? I guess it has smth to do with the lock icon on the image :)

I'm somewhat confused about how it all works.

Answer 1

Vault, like any other software, has the concept of a super user. That concept is called the "root token" (not be confused with a root key, we'll come back to that).

A full lecture on Vault internal architecture is way beyond the scope of a StackOverflow answer. Here is my attempt to clear up the confusion, leaving the details for you to explore.

Two takeaway to begin with:

• Vault stores a strong hash of the root token (if it is still valid)
• Vault does not store shards at all, anywhere, ever.

Super user access (aka root token)

When Vault is initialized, it will give you two set of data:

1. The almighty powerful root token
2. A number of Shamir shards (5 is the default, 1 in DEV mode)

The inital root token is always given out in plain text, never encrypted. It is meant to be used right away to perform the initial configuration.

Best practice is to use the root token only to:

1. Create a very powerful policy (let's call it almost-root)
2. Setup at least one authentication method
3. Bind one account of the authentication method to the almost-root policy
4. Revoke the root token (with vault token revoke -self)

At this point you can be almost-root, which should be behind some two factor auth, strong audit, and have a limited validity period (30 minutes), etc. Your CI/CD is often able to be almost-root.

Now let's say that for security reason, the almost-root policy does not allow to change the audit devices, adding or removing them.

To change the audit device configuration, you need to get a new root token. To get one, you must generate one from the shards you got at initialization time. It is a security measure. No one can become root on his own. You must collude with other which decreases the payout of a wrongdoing and increases the risk to get caught.

But let's move on and talk about generating a root token.

Shards and getting the root token

Vault will give you the shards in plain text. Shards are points on a Shamir curve.

When you initialize Vault, you should send public keys so that the shards come out encrypted. You can provide public keys on the command line (as file names) or with keybase.io aliases.

You can then safely distribute one shard per "shard holder". Have them decrypt and test their shard right away by generating a root token and revoking it right away. You don't want to find out that a user misplaced his private key or whatever.

You must test the shards on a regular basis. People come and go, computer crash. When you get close to your threshold, you must rekey to get new shards to new "shard holders". If your enterprise has a physical safe, consider generating enough extra shards to generate a root token, decrypt them, store them on a CD and print them out, and put all of that in a sealed enveloppe in the safe.

Generating a root token should be rare, but we use actually use it at the end of every sprint to generate an almost-root token that we give our CI/CD tool. That means that in day to day use, nobody has super powers in Vault. Define "super powers" to fit your operational reality.

Internal encryption

So the root token does not participate in the encryption at all, or else the system would stop when the root token is revoked. And you can't have 3 or more shard holders around for every decryption Vault does (and it can do a lot).

So Vault has an internal encryption key that is encrypted with key material outside of Vault. Suffice to say that it is either:

• Encrypted with a key made from the shards
• Encrypted with a KMS of HSM

When Vault starts, it cannot read its own storage, whatever storage backend you use. It is behind a cryptographic "barrier". You can give Vault credentials to your KMS so that it can decrypt the Key Encryption Key (KEK) that will give it its internal encryption key. That process is called auto-unseal.

Where do you store the Vault credentials to auto-unseal with a KMS or HSM you might ask? Good question, you have to start somewhere. Maybe you set your cloud policy to allow a given pod running Vault to access your KMS without a password.

If you can't setup auto-unseal, Vault will start in sealed state. You must enter a given number of shards (3 is the default) to allow Vault to generate that KEK and unseal itself. It will run like that until restarted or manually sealed.