Securely deploying secrets to boxes during provisioning

Question

During a fully automated deployment on physical hardware (using pxeboot / preseed / kickstart to install and configure a minimal system for bootstrapping) we need to deploy certain secrets - such as our Chef validation key, and a secret key for accessing secure application data.

In our VM environments we can pre-bake images with these already in so when the node is deployed it already has the files locally and they never have to cross the network, however for physical servers where we're building straight from the netboot images we don't have the ability to do this.

I'm trying to come up with both a secure and automated way of deploying these secrets out to servers that are provisioned, in a way that we know they could not end up in the wrong place.

Our provisioning environment has knowledge of all our physical servers and wether they've been provisioned or not - we create node definitions in our Chef server which include the MAC address of the primary interface used for PXEBooting, then set a flag on the node once it's been provisioned. We use this to only offer up PXEBoot files to nodes that have not yet been provisioned, and so we can specify in our node definitions the release / os / roles of the boxes.

Our current working idea is a very simple web service that (over HTTPS) would serve the secrets to nodes which have not yet been provisioned. However this has a number of drawbacks and issues I can see. My primary concern is our only protection is MAC address validation, and MAC addresses can be spoofed (I'm strongly of the opinion you should never consider your internal network secure, though that's probably a different discussion).

Our current provisioning environment is Ubuntu 12.04 and 14.04 with PXEboot / Preseeding on dedicated provisioning servers - though we could adapt other solutions to fit our environment.

I'm mainly concerned about a couple of global secrets that all nodes need access too, our ideal end goal is to not have these global secrets and have per-node secrets, I'm currently investigating a tool to accomplish this but for the moment we have to deal with the current setup.

So how are people securely, automatically distributing global secrets in provisioning environments? Any advise or pointers for ways to do it?

Answer 1

Pre-OS-boot there's not a lot of smarts alive other than the BMC (the little subsystem on the motherboard that implements IPMI/iDRAC/iLO/etc.), but it might be enough, depending on how much you want to trust it.

You could write a key to it (over the network using an encrypted IPMI 2.0 channel, or if that's not an option hopefully over a management LAN or VLAN (bleah)) during pre-boot sequence, and then have the server-to-be boot a vanilla/stub OS that simply boots an OS and contains no secrets or confidential stuff inside.

After booting the proto-server can communicate to its own BMC (root doesn't need auth locally), read the key using ipmitool or whatever, and then talk back to the mothership with the keys read providing the assurance that it's who it says it is; then final provisioning may begin. Using this (freely admitted :)) hack you could also use individual keys for each system.

Possible locations for storage of keys on the BMC include the MAC address (~48 bits of key space per ethernet port should be enough for anyone (sic!)), the SNMP community string, etc. All could be reset to reasonable values after the initial provisioning, and are easily modified.

Implementing this is modestly simple, with a script to generate keys on a per-server-basis, an ipmitool command to set the appropriate keys on the targets, a script in the little basic OS image that reads the local keys and uses them to authenticate itself to the mothership, and then the final provisioning. More complex than usual provisioning undoubtedly, but sometimes you have to pay for peace of mind.