AA | Design ideas to verify the initdata

[Xynnn007]

Background

Currently, we are promoting the initdata mechanism in the community. The core idea is to leverage initdata field (HOSTDATA for SNP, MRCONFIGID for TDX, etc.) to bind the hash value of any data injected into the guest. One of the most critical design is how AA exposes an interface for kata-agent to verify the binding between the hash of initdata plaintext and the corresponding initdata field inside TEE evidence.

Overall, we should take a way to do the following things:

1. launch AA by being forked in kata-agent's main function or by systemd.
2. receive the initdata plaintext/hash into AA from kata-agent
3. AA gets the TEE attestation report. This process must be without network, as AA is a service launched at a very early stage in guest where network is not prepared. The codes for TDX and SNP
   are finished by @danmihai1 perfectly in https://github.com/kata-containers/kata-containers/pull/8469/files#diff-df928933d70fb4e5616a6ecb3d8a1340adbf328dd3d6973b3a00ce5c75aae23aR1 which we can reuse in AA.
4. AA compares the hash of the initdata plaintext and the one inside evidence.

The step 2 is now to be designed. This issue aims to collect ideas upon this.

@fitzthum @danmihai1 and I have been discussing this and let's continue publicly.

Design Concerns and Different Ways

There are some design concerns

1. Although TDX, SNP, CCA, etc have initdata field like HOSTDATA, MRCONFIGID, s390x and AMD SEV do not.
2. The KBS url, KBS public key together with some other parameters should be part of AA's configuration file. Those are all things provided via initdata.

Thus, we need a mechanism

1. Allow s390x and SEV to not verify initdata without blocking AA from launching. While on other platforms AA should panic/raise error if initdata is not matched.
2. The configurations should be correctly configured and used if initdata binding check passes.

A. Initdata as a launch parameter

Initdata check will be part of AA's launch parameter.

attestation-agent --config-file conf.json --initdata MDAwMDAwMAo=

The parameter --initdata item is optional. If it is given, it will check the binding against an evidence report as soon as AA is launched.

For s390x/sev, AA will do nothing but only raise a warning On XXX platform initdata is not support, thus ignore --initdata parameter when it is given.

This is difficult for systemd scenarios, as we should have a caller before AA that knows the initdata and the configuration of AA. Also, if the AA fails to start, the caller (e.g. kata-agent) cannot know that.

B. Initdata as an AA API parameter

Let's temporarily call the api CheckInitdata.

1. After receiving the initdata, kata-agent will call CheckInitdata of AA to check the binding.
2. After checking the binding, extract AA's configuration file and update the configuration dynamically by calling new UpdateConfigurations of AA.

cc @jiazhang0 @jialez0

[mkulke]

One of the most critical design is how AA exposes an interface for kata-agent to verify the binding between the hash of initdata plaintext

Is kata-agent verifying Hash(InitData) == Hostdata really a crucial aspect of the InitData concept? isn't the TEE report and hence Hostdata essentially untrusted before remote attestation?

[Xynnn007]

Is kata-agent verifying Hash(InitData) == Hostdata really a crucial aspect of the InitData concept? isn't the TEE report and hence Hostdata essentially untrusted before remote attestation?

Nice question. About this Tobin and Dan gave me an attack scenario, which summrized by Tobin
'''
what dan and I were saying is that the Kata Agent / AA must check the binding of the policy and the initdata before using the policy i.e. when the policy is set. it can't wait until remote attestation to do it.. if we wait until remote attestation to check that the hash of the policy matches the initdata, there is a possible attack.
for the attack, the goal of the attacker would be to create a malicious guest that has the correct measurement so that the guest could retrieve secrets and then steal them. here's how you could do that:

1. host sets the HostData with the correct policy.
2. host starts guest and sends the agent a different policy that allows all endpoints to be open
3. host uses one of these unprotected endpoints to break into the guest
4. the malicious guest performs a remote attestation. since the host data is correct, the attestation is successful

this can be prevented at step 2. before the kata agent uses the policy, we check the binding to the evidence. if the check does not pass, we don't use the new policy.

this connects to something i was saying at the beginning of the meeting about the design of the AA endpoint that is going to check the binding. this endpoint must be available very early (before remote attestation) and it can't rely on any configuration that will be provisioned via the InitDat/Policy. What I envisioned is a very simple endpoint that detects which platform is running and then checks that the input matches the report.
'''

[mkulke]

Thanks, that's an interesting scenario. If I understand it correctly this means the kata-agent is applying an untrusted policy initially. But why should the kata-agent do that? Shouldn't everything be locked down and only endpoints selectively opened using a verified policy?

[Xynnn007]

I am not a master of this. Let me try to explain.

Now the policy is injected by pod yaml's annotation, and this will happen in a same context where the pod VM is created. During the same time, network set-up APIs will also be called and filtered by the policy. So before we can verify the policy by remote attestation, the policy is already in-use within a guest where the network is not prepared.

If we want a way to use a new policy after attestation and before that a strict & verified policy is used, we might need to store the policy in KBS and specify the resource id of KBS in the initdata/launch parameter of kata-agent.

Hi @danmihai1 please correct me if any.

[mkulke]

network set-up APIs will also be called and filtered by the policy

ok, this might be the key to understanding the reasoning behind that assertion, as the policy doesn't just apply to container workloads but also to the "infrastructure tier" in the guest to some degree.

Hrm yeah it's tricky, kata cannot be fully-locked down fully, since SetPolicy(), CreateSandbox(), CreateContainer() cannot be locked down really, since this is how we receive the pod spec, the policy and the init-data to bootstrap remote-attestation.

it just feels odd, it appears like there is untrusted code (unverified kata-agent + OPA bins) verifying an untrusted policy (received via SetPolicy) using untrusted data (field in an unverified TEE report). And we can only state retroactively, after verifying each of those components via remote attestation, that the first verification was valid (some kind of higher-order verification ♻️)

[mkulke]

I remembered, this was also discussed in the peerpod context a while ago and @katexochen suggested to only allow SetPolicy() initially. Is this something that would work?

confidential-containers/cloud-api-adaptor#1369 (comment)

[katexochen]

what dan and I were saying is that the Kata Agent / AA must check the binding of the policy and the initdata before using the policy i.e. when the policy is set. it can't wait until remote attestation to do it.. if we wait until remote attestation to check that the hash of the policy matches the initdata, there is a possible attack.

Notice that this is not true in situations where we can do measurements at runtime (vTPM, for example with peerpods). In this case, we can policy when it is set, and we don't need to know the policy value upfront. Even if the attacker sets a malicious policy, they cannot influence the measurement of the policy that is done before the policy takes effect. A verifier can then later check the measurement and the attacker cannot prevent the verifier form discovering the malicious policy.

However, I agree that in situations where we only have an initial launch measurement, we must validate the policy hash before the policy takes effect.

[fitzthum]

I lean towards option B because it avoids the difficulty with systemd and in the non-systemd case it makes the changes needed in kata simpler.

One potential drawback with option B is that adding a dynamic configuration update endpoint might make the attack surface of the AA larger. I don't think it creates any obvious security issues, but it might make it a little bit easier for an attacker to tamper with the AA. Maybe we should only allow this endpoint to be called once?

The tricky part is making sure this will still work with static platforms like SEV and s390x. For other guest components, this will be easier. For example, the CDH will simply expect a config file in a certain location. Either this will be added to the initrd ahead of time or provisioned dynamically via InitData. This won't work for the AA because the config contains more dynamic data.

Currently we configure the AA via the AA_KBC_PARAMS kernel parameter. We probably want to keep this mechanism for the static platforms. I would like to minimize the amount of overhead in the Kata Agent so maybe we can add a check to the AA. For instance, when the AA starts, it could check the kernel command line to see if AA_KBC_PARAMS is set. If it is, the config would be built from there. If not, the config would be expected via the UpdateConfiguration endpoint. It's a little weird to have these two different code paths, but I'm not sure how to avoid it.

Another question is how we make this work for peer pods. I would like to keep the peer pods case as similar to the normal case as possible. One way to do this is to make the CheckInitData endpoint work with runtime measurements. For platforms with runtime measurements, the endpoint would simply extend one of the PCRs with the hash of the policy. I think if we did this peer pods could just call SetPolicy/SetInitData from the host. Maybe they actually want to get their configurations from somewhere else though? cc @mkulke @bpradipt

[mkulke]

One way to do this is to make the CheckInitData endpoint work with runtime measurements.

If I understood it correctly the endpoint would imply a fn check_init_data(init_data_hash: &[u8], evidence: &Evidence) on AA's Attester trait. the TEE specific implementations would perform an assertion like init_data_hash == evidence.host_data for SNP. That fn would be local to the TEE, since only after that we'd be able to bootstrap a connection to an attestation-service using parameters from init_data.

Intuitively I'd think such a fn would be implemented as a no-op on TEEs which are not able to perform this assertion, because there is no field in the HW evidence that would hold the initdata hash.

On TEEs that are able to perform runtime measurements (TPM, tdx via rtmr?) init_data_hash would be part of the TEE evidence (to be verified in remote attestation). So, in theory there could be an implementation that performs init_data_hash == evidence.runtime_measurement. Not sure if that assertation yields anything?

[fitzthum]

What I envision is a function fn check_init_data(init_data_hash: &[u8]) where

SNP/TDX
Gets the report, compares host_data to init_data_hash.
For TDX we could use an RTMR, but I think it's best to keep these implementations similar.

SEV/s390x
Return an error. Init data is not supported on these platforms. Maybe in the future we could try to support some kind of signature for init data hashes where we could pre-provision or inject the public key. This is low priority.

vTPM
On platforms that don't have have already consumed the init data field but that provide vTPM, we extend some PCR with init_data_hash.

Maybe check_init_data isn't the best name since it would actually do a couple different things. It's also possible that we could have this distinction happen at a different level and simply have peer pods use a different codepath that calls the runtime measurement update directly.

[Xynnn007]

Maybe check_init_data isn't the best name since it would actually do a couple different things. It's also possible that we could have this distinction happen at a different level and simply have peer pods use a different codepath that calls the runtime measurement update directly.

Right. IMO, for scenarios where vTPM is not provided, kata-agent will try to call check_init_data of AA. For scenarios where vTPM is usable, kata-agent will try to record the event by runtime measurement with vTPM. This API would also be provided by AA. See #392 (comment) cc @jialez0

This logic of checking the existance of vTPM might be deterimined by kata-agent in runtime to avoid complex compilation features.

[fitzthum]

This logic of checking the existance of vTPM might be deterimined by kata-agent in runtime to avoid complex compilation features.

I am a bit wary of adding attestation-specific code to the kata agent. The vTPM case is mainly for peer pods for now. I don't know what flow they want to use for provisioning config stuff.

[Xynnn007]

I am a bit wary of adding attestation-specific code to the kata agent. The vTPM case is mainly for peer pods for now. I don't know what flow they want to use for provisioning config stuff.

You are right. The requirements are

1. AA should have straightforward API semantics. Up to now, there should be get_evidence(), get_token(), extend_runtime_measurement() and proposed check_init_data().
2. There should not be any attestation related code inside kata-agent.

After some thinking I realize that we just need to do two things.

1. Add the check_init_data() API to AA, and kata-agent will always call it.
2. Add extend_runtime_measurement() call to some components like process-user-data (functionally) in peerpod.

The difference between peerpod and kata-cc is the whole design ways. kata-cc uses initdata field inside evidence to bind the integrity of initdata and therefore should leverage AA's check_init_data() to prevent untrusted host from pollute the initdata.

Peerpod uses vTPM's runtime measurement ability to bind the integrity. So extend_runtime_measurement() should be used to extend PCR register to record the integrity.

We can always let kata-agent call check_init_data() to AA. Like @mkulke once said

Intuitively I'd think such a fn would be implemented as a no-op on TEEs which are not able to perform this assertion, because there is no field in the HW evidence that would hold the initdata hash.

On vTPM based platforms, it hints that a component like process-user-data has already bound the integrity of initdata things to vTPM runtime measurement. This is also based on an assumption -

1. If the current environment supports setting initdata/hostdata field, the initdata things is already injected by the mechanism. Like TDX/SNP/..
2. If the current environement does not support setting initdata/hostdata field, vTPM should be supported. Like azure-snp/...
3. If neither initdata nor vTPM is supported, like SEV-ES and IBM SE. They have their special methods to inject information, e.g. secure module for SEV-ES and contract for SE, where we might not need to care about in this context.

Did I ignore anything important?

This way, we do not break/extend any semantics of the APIs.

[mkulke]

@Xynnn007 thanks for the write up, that's a good summary. Logically this all makes sense to me, in the concrete implementation aspects we'll have to see, what we have to adjust. At the moment process-user-data (PUD) is a one-off service unit required by kata-agent (peerpod images use systemd) and kata-agents spawns AA as a subprocess.

So PUD wouldn't be able to call AA, since there's no AA process yet. I'm not entirely sure about the plans for subprocessing attestation-agent in kata#main (whether we want to keep it or have some sort of init system), so it might not be a problem in the future.

kata-agent depends on PUD, because PUD is templating the aa_kbc_param string into the agent-config. Even if we start AA independently from kata-agent and AA has its own configuration file. If we use PUD to provision the config (UserData => AA.cfg), there would still be a cyclic dependency (PUD => AA => PUD).

So, how and if peerpod would be able to use the extend_runtime_measurement() endpoint in a vTPM setup, we have to see. but I don't think it's a crucial question for this design. We can just assume that some component is measuring the init_data hash into a PCR, so it will be a legitimate part of the evidence and can be treated similar as a HOSTDATA/MRCONFIGID field on the atttestation-service/verifier side.

[Xynnn007]

kata-agent depends on PUD, because PUD is templating the aa_kbc_param string into the agent-config. Even if we start AA independently from kata-agent and AA has its own configuration file. If we use PUD to provision the config (UserData => AA.cfg), there would still be a cyclic dependency (PUD => AA => PUD).

Right. Seems that the cyclic denepdency could be handled by adding an extra API UpdateConfiguration() for AA on both kata-cc and peerpod. That means that AA should be launched at a very early time with default configs. At that time, AA only provides ability without network like check_init_data() and extend_runtime_measurement(). Then, PUD/kata-agent will call extend_runtime_measurement()/check_init_data() and update the real configuration to AA, thus providing AA more abilities like remote attestation. The above assumption is AA is launched by systemd, where peerpod could easily apply.

The reason why I mentioned PUD call AA to leverage PCR, is a) AA already has the ability to do runtime measurement extending b) putting all attestation functionalities to one component seems more "clean".

[mkulke]

The reason why I mentioned PUD call AA to leverage PCR, is a) AA already has the ability to do runtime measurement extending b) putting all attestation functionalities to one component seems more "clean".

Yes, as we have the endpoint, we should use it if possible, I agree.

[huoqifeng]

network set-up APIs will also be called and filtered by the policy

ok, this might be the key to understanding the reasoning behind that assertion, as the policy doesn't just apply to container workloads but also to the "infrastructure tier" in the guest to some degree.

Hrm yeah it's tricky, kata cannot be fully-locked down fully, since SetPolicy(), CreateSandbox(), CreateContainer() cannot be locked down really, since this is how we receive the pod spec, the policy and the init-data to bootstrap remote-attestation.

it just feels odd, it appears like there is untrusted code (unverified kata-agent + OPA bins) verifying an untrusted policy (received via SetPolicy) using untrusted data (field in an unverified TEE report). And we can only state retroactively, after verifying each of those components via remote attestation, that the first verification was valid (some kind of higher-order verification ♻️)

I doubt if the init_data also includes the KBS service url, how can we assure the remote attestation is trusted? The kbs could also be a malicious endpoint.

[mkulke]

I doubt if the init_data also includes the KBS service url, how can we assure the remote attestation is trusted? The kbs could also be a malicious endpoint.

I would assume the situation for setting a hash of the KBS endpoint to the report's hostdata field is logically not very different from passing the endpoint to the guest via kernel cmdline. In both cases the KBS endpoint (directly or as a derived hash) is part of a signed TEE report.

Now, a malicious, privileged actor could tamper with either one of those and inject a new KBS endpoint. That KBS endpoint could fake remote attestation. Consequences: The fake KBS is not able to provide secrets, but it might deliver the wrong ones. It would seem that a workload is running encrypted in a TEE, when it's not.

The former (malicious secret) is an issue that deserves attention IMO, because there's a practical use case that comes to mind with Azure's SAS or AWS' Presigned URLs. Those are often used as an alternative to specific IAM privileges enabling a workload to push/retrieve data from an Object Store.

The latter (masquerading a non-encrypted workload as a running in a TEE) is a fundamental issue with CoCo, i think. With confidential pods a kubernetes user cannot trust what the control plane is reporting anyway. The consequence might be that only a workload whose execution is depending on a secret stored in a trusted KBS can be trusted itself (like an encrypted image).

This topic is discussed at length in Tobin's gist, but I'm afraid there's no clear answer yet.