Remote Attestation
A host connects to your network. It looks like part of your fleet, but is it really? It runs your software, or at least enough of it to start serving traffic. It quacks like a duck. But what’s on the inside?
I’m surprised ChatGPT agreed to make that one.
But seriously. How do you know that your data isn’t sitting on a compromised stack? How do you know that you aren’t trying to schedule workloads on an attacker controlled machine? How do you know any of your security controls are actually in place on a host? And once an attacker has taken over a host, there are endless places to hide and persist through upgrades and reboots.
Most setups have a lot of faith in hosts once they have been provisioned. Hosts are placed in the trust boundary and then just remain there, regardless of whatever has happened in the mean time. If they even should have been there in the first place.
Enter the wonderful world of remote attestation which (mostly) solves this problem. Using a TPM, we can remotely, cryptographically prove a couple of things:
The host has the expected hardware
The host has the expected firmware
The host has the expected kernel and init image
The host has the expected root filesystem (depends on the setup)
The host passed arbitrary checks that we define
This means that after a reboot, we know the exact state a host is in. Without extensive physical modifications and tampering, a fresh boot either puts us in a trusted state or fails and breaks trust in the host.
The measure boot cycle is different then trusted boot, it relies on signed measurements, not signed artifacts like secure boot. It offers a much bigger coverage surface at the cost of considerable complexity. Even malicious, signed, drivers and rootkits can be caught.
But what’s the point of measuring if we don’t block boot like secure boot does? What does measurement get us?
This gets into how you trust hosts. You can use your TPM to encrypt all or part of your root-fs (where all your keys and data are), so that a host is functionally inoperable without the measured boot succeeding. Your RA can refuse to issue certificates to a host that doesn’t provide correct measurements (you are using mTLS right?). You can even back TLS x509 certs with the TPM so an incorrectly rebooted host can’t authenticate over mTLS at all. This offers extremely strong guarantees about the initial state of hosts, so you can trust that your workloads and data are in an environment you trust.
This isn’t a perfect solution. Physical attacks (like memory taps) are still a problem. But coupled with robust EDR and a good LSM policy, you have no closed the door on a huge number of issues, in particular nasty kernel and driver supply chain attacks that bypass all of your nice userland defenses.
But what about attacks after boot?
That’s your EDR’s problem. Trusted boot provides the bedrock to build a bunch of other primitives on top of. Including cryptographic proof your EDR is installed and running (at boot), immutable filesystems (verified at boot), signed upgrades, confidential computing, etc. Without it you can’t trust your hosts themselves and can’t make further security guarantees. Houses built on sand and all that.
Unfortunately, remotely measuring every step of your boot process, for every host in your organization, is a tremendous undertaking. It goes into how you build and distribute firmware, init images, kernels. On the plus side, it will give you a chance to clean up a bunch of legacy cruft and also probably reveal a bunch of supply chain issues you weren’t thinking about.
TPM magic more or less. TPMs expose a limited interface of cryptographic operations to the host, and that interface is locked down sufficiently that it controls your access to cryptographic material. They are also resistant to physical attacks, but the main function is to mitigate software attacks, not physical. Even kernel code can’t dig into a TPM and extract private material.
The TPM comes with a set of PCRs (Platform Configuration Registers). Their only function is to store hashes. They only go forward, you can measure new hashes into a PCR and it hashes the previous value with the new one, yielding a unique result. No going backwards. Different artifacts get measured into the different PCRs during boot and init, and similar to secure boot, each phase measures the next phase.
Breaking this chain anywhere will taint the PCR values.
The TPM cares about these values. You can do a couple thing based on them:
TPM sealing: material can be stored in the TPM and only unlocked if the PCRs (the ones you chose) check out.
TPM quotes: the values can be signed with a key resident to the TPM. Your RA knows the providence of that key (more in a moment) and this offers strong proof that the measurements are correct.
The full ritual is outside the scope of a blog post like this (ask Claude), but it goes something like this:
When a host is first be provisioned the TPM has an Endorsement Key (EK) which is a decryption key burned in by the manufacturer. The EK comes with a x509 cert signed by the manufacture’s PKI. So the EK proves the TPM is legit.
You then mint an Attestation Key (AK), derived from the EK. The AK is a restricted signing key, it can only sign TPM quotes, nothing else (what would the point of this be if you could make your own quote and sign it) and you generally persist the AK for subsequent reboots.
But wait, how do we know that an AK is tied to an EK? The EK is a decryption key, it doesn’t sign anything. The attester just gets two public keys, the EK and AK public keys and the EK x509 leaf cert. The full answer involves more crypto math then I am comfortable trying to explain. TLDR the verifier issues a challenge: encrypting a credential with the EK public key and sending it back. The device decrypts the credential with the EK private key (which never leaves the TPM), and sends the credential to the attester, proving that it is talking to a device that has control of the EK private key.
The AK can only sign quotes, and the attester has a set of permissible PCR values for each register. So an AK proves the host booted the way we expect, the “measured boot” side of the story. But need a third key, the LDevID (Locally-scoped Device ID), which represents a node identity.
The LDevID is a child of the AK. Importantly, the LDevID can sign anything, not just quotes. Anything signed by the LDevID is proven to have been signed with this specific TPM. Importantly, since it can sign anything we need to restrict when signing can take place — otherwise an attacker can gain control of it. Frequently it is sealed to golden PCR values, so a reboot into a bad state cannot use it.
Once the whole ritual is done the host can actually prove itself to be legitimate. This does not ensure runtime security, that part is on you. This is only about boot. If this is all set up correctly, attacks will not survive a reboot. Or at least can only brick the host.
TPM sealing runs into problems with upgrades. New firmware has different measurements and will break attestation. The solution is to authorize a policy (TPM2_PolicyAuthorize), signed by an AuthKey, that authorizes the TPM to accept new PCR values. The AuthKey can be held by the node itself, in which case it needs to vet updates and determine whether or not to accept them, or an authority hold it and distributes the public key to the device/TPM. It just depends if you want to validate upgrades on device or elsewhere in your infra.
If your infra consistently enforces mTLS, you can reject hosts that were not attested, since the RA won’t give them certs. If you use TPM backed certs, tampered hosts naturally lose access to the rest of production.
Every host has providence. Data signed by the LDevID (whether TLS or something else), you can prove is from a specific host.
Workloads can refuse to run on hosts that were not attested. Your scheduler can demand cryptographic proof before allowing jobs to run on a host. Your jobs themselves can challenge a host before pulling in data.
Like I said above, if you are confident in your boot security you can build your runtime security appropriately. In the next post I’ll get into dm-verity and immutable root images with systemd.
