Support for Arm CCA guest firmware v3 (RMM-v1.0-rel0)

[samimujawar]

Description

This v3 series enables the Arm Confidential Compute Architecture (CCA)
support for the Kvmtool guest firmware and is aligned with the ARM CCA
RMM 1.0-rel0 specification.

In addition to the RMM 1.0-rel0 specification updated this series has
been rebased with the latest edk2 & edk2-platforms codebase and the
intention is to integrate the Arm CCA support in ArmVirtPkg and enable
the guest firmware support for Realms.

Summary of updates in this v3 Series:

1. Updates to align with RMM 1.0-rel0 specification.
2. Add support to parse the Kvmtool guest VM Device Tree
   and populate the memory map accordingly.
3. Check if the MMIO range is for a protected Realm device
   and if so, skip configuring the shared attribute for the
   MMIO region.
4. Add check to conditionally install the IORT table if
   a GIVc3-ITS is present.

Summary of updates in the previous v2 Series:

1. Variable emulation support patches that we part of v1 series
   are already merged, hence dropped from this series.
2. SetMemoryRegionAttributes() was dropped in the upstream
   code. Therefore, introduced SetMemoryProtectionAttribute()
   to configure the top bit of the Realm IPA space which is
   used as the protection bit
3. The patch to add the APRIORI Dxe ArmCcaDxe has been dropped.
4. Dropped patch that configured PcdMonitorConduitHvc as a
   dynamic PCD, and introduced ArmVirtMonitorLib, a new
   instance of the ArmMonitorLib that reads the conduit to
   be used from the FDT.
5. Bug fixes to correct the size of IMM field in RSI Host
   Call arguments, and to correct the RSI Version mask
6. Patches 32 to 43 include updates to the firmware support
   to RMM specification v1.0-EAC5.
7. Minor optimisations, e.g. to cache the current world value.

Introduction

Arm Confidential Compute Architecture (CCA)

Arm CCA is a reference software architecture and implementation that
builds on the Realm Management Extension (RME), enabling the execution
of Virtual machines (VMs), while preventing access by more privileged
software, such as hypervisor. Arm CCA allows the hypervisor to control
the VM, but removes the right for access to the code, register state or
data used by VM.

More information on the architecture is available here [1].

        Realm World     ||    Normal World   ||  Secure World  ||
                        ||        |          ||                ||
 EL0 x---------x        || x----x | x------x ||                ||
     | Realm   |        || |    | | |      | ||                ||
     |  VM*    |        || | VM | | |      | ||                ||
     |x-------x|        || |    | | |      | ||                ||
     ||       ||        || |    | | |  H   | ||                ||
     || Guest ||        || |    | | |      | ||                ||
 ----||  OS   ||--------||-|    |---|  o   |-||----------------||
     ||       ||        || |    | | |      | ||                ||
     |x-------x|        || |    | | |  s   | ||                ||
     |    ^    |        || |    | | |      | ||                ||
     |    |    |        || |    | | |  t   | ||                ||
     |+-------+|        || |    | | |      | ||                ||
     || REALM ||        || |    | | |      | ||                ||
     || GUEST ||        || |    | | |  O   | ||                ||
     || UEFI  ||        || |    | | |      | ||                ||
     |+-------+|        || |    | | |  S   | ||                ||
 EL1 x---------x        || x----x | |      | ||                ||
          ^             ||        | |      | ||                ||
          |             ||        | |      | ||                ||
 -------- R*------------||----------|      |-||----------------||
          S             ||          |      | ||                ||
          I             ||      x-->|      | ||                ||
          |             ||      |   |      | ||                ||
          |             ||      |   x------x ||                ||
          |             ||      |       ^    ||                ||
          v             ||     SMC      |    ||                ||
      x-------x         ||      |   x------x ||                ||
      |  RMM* |         ||      |   | HOST | ||                ||
      x-------x         ||      |   | UEFI | ||                ||
          ^             ||      |   x------x ||                ||
 EL2      |             ||      |            ||                ||
          |             ||      |            ||                ||
 =========|=====================|================================
          |                     |
          x------- *RMI* -------x

 EL3                   Root World
                       EL3 Firmware
 ===============================================================

Where:
RMM - Realm Management Monitor
RMI - Realm Management Interface
RSI - Realm Service Interface
SMC - Secure Monitor Call

RME introduces two added additional worlds, "Realm world" and "Root
World" in addition to the traditional Secure world and Normal world.
The Arm CCA defines a new component, Realm Management Monitor (RMM)
that runs at R-EL2. This is a standard piece of firmware, verified,
installed and loaded by the EL3 firmware (e.g., TF-A), at system boot.

The RMM provides a standard interface Realm Management Interface (RMI)
to the Normal world hypervisor to manage the VMs running in the Realm
world (also called Realms). These are exposed via SMC and are routed
through the EL3 firmware.

The RMM also provides certain services to the Realms via SMC, called
the Realm Service Interface (RSI). These include:

• Realm Guest Configuration
• Attestation & Measurement services
• Managing the state of an Intermediate Physical Address (IPA aka GPA)
  page
• Host Call service (Communication with the Normal world Hypervisor).

This patch series aligns with the RMM v1.0-rel0 specification, and
the latest version is available here [2].

The Trusted Firmware foundation has an implementation of the RMM -
TF-RMM - available here [4].

Implementation

This version of the Realm Guest UEFI firmware is intended to be
used with the Linux Kernel stack[7] which is also based on the
RMM specification v1.0-rel0[3].

This release includes the following features:
a) Boot a Linux Kernel in a Realm VM using the Realm Guest UEFI
firmware
b) Hardware description is provided using ACPI tables
c) Support for Virtio v1.0
d) All I/O are treated as non-secure/shared
e) Load the Linux Kernel and RootFS from a Virtio attached disk
using the Virtio-1.0 PCIe transport.

Overview of updates for enabling Arm CCA

The Arm CCA implementation is spread across a number of libraries
that provide required functionality during various phases of the
firmware boot.

The following libraries have been provided:
i. ArmCcaInitPeiLib - A library that implements the hook functions
in the PEI phase
ii. ArmCcaLib - A library that implements common functions like
checking if RME extension is implemented and to configure the
Protection attribute for the memory regions
iii. ArmCcaRsiLib - A library that implements the Realm Service
Interface functions.

A NULL implementation of the ArmCcaInitPeiLib and ArmCcaLib is also
provided for platforms that do not implement the RME extensions.

Additionally, the following DXE modules have been provided to implement
the required functionality in the DXE phase.
i. RealmApertureManagementProtocolDxe - A DXE that implements the
Realm Aperture Management Protocol, used to manage the sharing
of buffers in a Realm with the Host
ii. ArmCcaIoMmuDxe - A driver which implements the EDKII_IOMMU_PROTOCOL
that provides the necessary hooks so that DMA operations can be
performed by bouncing buffers using pages shared with the Host.

Arm CCA updates in PEI phase

For supporting Arm CCA two hooks have been added in the PrePi module:
i. An early hook to configure the System Memory as Protected RAM
ii. A second hook after the MMU is initialised to perform the
remaining CCA initialisations like reading the Realm Config
to determine the IPA width of the realm, configuring the
Protection attribute for the MMIO regions, etc.

These hook functions are implemented in ArmCcaInitPeiLib. A NULL
version of the library has also been provided for implementations
that do not have the RME extensions.

Additionally, the ArmVirtMemInfoLib has been updated to implement
a platform specific hook function ArmCcaConfigureMmio() that can
configure the protection attribute for the MMIO regions for the
platform.

   +=====+
   |PrePi|
   +=====+
      |
      _ModuleEntryPoint()
      ===================
              |
              DiscoverDramFromDt()
              |
              +--> ArmCcaInitPeiLib|ArmCcaConfigureSystemMemory()
                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                             |      // configure System Memory
              ----------------      // as Protected RAM.
              |
             ...
              |
      --------
      |
      CEntryPoint()
      |
      PrePiMain()
      ===========
          |
         ...
          |
          ProcessLibraryConstructorList()
          |
          MemoryPeim()
          |
          ArmCcaInitPeiLib|ArmCcaInitialize()  // Perform Arm CCA
          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  // initialisations,
                   |                           // like reading the
                   |                           // Realm Config, etc.
                   |
                   ArmVirtMemInfoLib|ArmCcaConfigureMmio()
                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                          |   // Configure Protection attribute
                   --------   // for the MMIO region.
                   |
          ----------
          |
         ...
          |
        +===+
        |DXE|
        +===+

Building the UEFI firmware

a. Set up the development environment
Follow the steps as described in
https://github.com/tianocore/edk2-platforms/blob/master/Platform/ARM/Readme.md

b. The source code for the Host and Realm Guest firmware can
be downloaded from [12].

c. Building the Host UEFI firmware for FVP Base RevC AEM Model
Follow the instructions in
https://github.com/tianocore/edk2-platforms/blob/master/Platform/ARM/Readme.md
to "Build the firmware for Arm FVP Base AEMv8A-AEMv8A model
platform" based on your development environment configuration.

Note: The same firmware binary can be used for both the Arm FVP
Base AEMv8A-AEMv8A and the FVP Base RevC AEM Model.

d. Building the Realm Guest UEFI firmware for kvmtool:
To build the kvmtool guest firmware, run the following commands:

   $build -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtKvmTool.dsc -b DEBUG
   $build -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtKvmTool.dsc -b RELEASE

The Kvmtool guest firmware binaries are at the following location:

   $WORKSPACE/Build/ArmVirtKvmTool-AARCH64/<DEBUG|RELEASE>_GCC5/FV/KVMTOOL_EFI.fd

Running the stack

To run/test the stack, you would need the following components:

i. FVP Base AEM RevC model with FEAT_RME support [5]
ii. TF-A firmware for EL3 [6]
iii. TF-A RMM for R-EL2 [4]
iv. Linux Kernel [7]
v. kvmtool [8]
vi. UEFI Firmware for Arm CCA [12].

Instructions for building the remaining firmware components and
running the model are available here [10]. Once, the host kernel
has finished booting, a Realm can be launched by invoking the
lkvm command as follows:

 $ lkvm run --realm \
   --restricted_mem \
   --measurement-algo=["sha256", "sha512"] \
   --firmware KVMTOOL_EFI.fd \
   -m 512 \
   --irqchip=gicv3-its \
   --force-pci \
   --disk <Disk image containing the Guest Kernel & RootFS>
   <normal-vm-options>

Where:

• --measurement-algo (Optional) specifies the algorithm selected for
  creating the initial measurements by the RMM for this Realm (defaults
  to sha256)
• GICv3 is mandatory for the Realms
• --force-pci is required as only Virtio-v1.0 PCIe transport is
  supported.

Links

[1] Arm CCA Landing page (See Key Resources section for various documentations)
https://www.arm.com/armcca

[2] RMM Specification Latest
https://developer.arm.com/documentation/den0137/latest

[3] RMM v1.0-rel0 specification
https://developer.arm.com/documentation/den0137/1-0rel0

[4] Trusted Firmware RMM - TF-RMM
https://www.trustedfirmware.org/projects/tf-rmm/
GIT: https://git.trustedfirmware.org/TF-RMM/tf-rmm.git
TAG: rmm-spec-v1.0-rel0

[5] FVP Base RevC AEM Model (available on x86_64 / Arm64 Linux)
https://developer.arm.com/Tools%20and%20Software/Fixed%20Virtual%20Platforms

[6] Trusted Firmware for A class
https://www.trustedfirmware.org/projects/tf-a/

[7] Linux kernel support for Arm-CCA
https://gitlab.arm.com/linux-arm/linux-cca
Linux Host branch: cca-host/v5
Linux Guest branch: cca-guest/v6
Full stack branch: cca-full/v5+v6

[8] kvmtool support for Arm CCA
https://gitlab.arm.com/linux-arm/kvmtool-cca
Branch: cca/v3

[9] kvm-unit-tests support for Arm CCA
https://gitlab.arm.com/linux-arm/kvm-unit-tests-cca
Branch: cca/v2

[10] Instructions for Building Firmware components and running the model, see
section 4.19.2 "Building and running TF-A with RME"
https://trustedfirmware-a.readthedocs.io/en/latest/components/realm-management-extension.html#building-and-running-tf-a-with-rme

[11] RFC series posted previously for adding support for Arm CCA guest firmware:
v2: https://edk2.groups.io/g/devel/message/117716
v1: https://edk2.groups.io/g/devel/message/103581

[12] UEFI Firmware support for Arm CCA
Host & Guest Support:
- Repo:
edk2: https://gitlab.arm.com/linux-arm/edk2-cca
edk2-platforms: https://gitlab.arm.com/linux-arm/edk2-platforms-cca
- Branch: 3223_arm_cca_rmm_v1.0_rel0_v3
- URLs:
edk2: https://gitlab.arm.com/linux-arm/edk2-cca/-/tree/3223_arm_cca_rmm_v1.0_rel0_v3
edk2-platforms: https://gitlab.arm.com/linux-arm/edk2-platforms-cca/-/tree/3223_arm_cca_rmm_v1.0_rel0_v3

---

• Breaking change?
  No
• Impacts security?
  No
• Includes tests?
  See ‘Running the stack’ section above.

How This Was Tested

See ‘Running the stack’ section above.

Integration Instructions

N/A

[kraxel]

In DXE ArmVirt already uses RNDR if available, via DxeRngLib -> EFI_RNG_PROTOCOL -> RngDxe -> BaseRngLib.inf

Using BaseRngLib.inf unconditionally isn't a good idea I think. There is too much hardware in the wild which has no RNDR support.

[kraxel]

Why is this in the Arm CCA PR? On a quick glance this looks like some independent change.

Also why put this into ArmVirt? We have other platforms like riscv which use device trees too and can probably use this code too, so I think OvmfPkg would be a better place for it.

[samimujawar]

Hi Gerd,

Thank you for reviewing the patch series and for the feedback.

Why is this in the Arm CCA PR? On a quick glance this looks like some independent change.

Yes, I agree this is an independent change and is a general improvement for normal VMs as well. However, this was introduced as this a requirement for a Realm VM, please see
43476c3 and therefore submitted along with this series.
However, please do let me know if you think I should submit this as a separate patch.

Also why put this into ArmVirt? We have other platforms like riscv which use device trees too and can probably use this code too, so I think OvmfPkg would be a better place for it.

I agree this can be utilised by other architectures as well and it can be moved to OvmfPkg. However, I think there may be some additional changes required for supporting other architectures, e.g. there is a pending series that should help pave the way for this change, see a1f2654

Since this patch is a subset of the FdtHwInfoParser, I think similar changes would apply here as well. Considering that, I think it would be good to pursue this effort once the RiscV support is enabled in Dynamic Tables.

Please do let me know if you think otherwise, and I will update this PR accordingly.

Regards,

Sami Mujawar

[kraxel]

Phew. Is there some overview how the hardware detection works in for Realm VMs? Apparently this is different from the usual qemu workflow, where qemu generates both FDT and ACPI tables, the guest firmware simply downloads them via FwCfg and installs them in guest memory.

So Realm VMs get a FDT only apparently (where does it come from?), then use DynamicTablesPkg to generate the ACPI tables inside the guest, correct? Where exactly in this workflow is this change needed? Parse the FDT and generate descriptions tables which DynamicTablesPkg can consume?

On riscv: Yes, sure, additinal changes will be needed to have the riscv firmware actually use this (and this is obviously beyond the scope of this PR).. But if this is something which is likely to happen we should merge the code into OvmfPkg right away instead of merging to ArmVirt and move over to OvmfPkg later on.

[jpbrucker]

I believe the ACPI table generation is specific to the ArmVirtKvmTool. If I remember correctly, ArmVirtQemu and ArmVirtCloudHv don't have it, so it's not (yet) a property of Realm VMs, only Kvmtool VMs. At the moment in my experimental Realm support for ArmVirtQemu, QEMU still generates ACPI tables in addition to FDT (though I haven't tested ACPI boot).

There is a larger question of whether we should create a single edk2 image for all Realm VMs, based on ArmVirtKvmtool but supporting QEMU, cloud-hypervisor and others. Given that Realm VMs have specific needs (initialization through RMM RSI, early attestation, lack of NV storage, reduced attack surface...) it would be nice to have all those things in a single image rather than porting them to each existing ArmVirt*. I've been meaning to investigate this but haven't found the time.

Having FDT->ACPI generation on this single image could also be useful for attestation. Given that firmware tables are measured (to ensure the untrusted host does not include harmful data in them), a verifier needs to be able to independently reconstruct the firmware tables, and it would reduce the efforts needed by verifiers to only measure one set of firmware tables rather than both FDT and ACPI. In QEMU the FDT is placed in memory, and ACPI tables are sent via FwCfg (whose address is found in the FDT). I believe CloudHV places both in memory and the project doesn't want to implement FwCfg.

This would require FDT to be able to represent everything ACPI can (at least for arm64 VMs), and some things may be missing. vCPU hotplug is a recent feature on Arm and I believe still requires ACPI at the moment. There might be others.

[kraxel]

One of the reasons qemu generates ACPI tables for the guest is to decouple firmware and qemu, i.e. qemu can add new features which require ACPI table updates without depending on firmware updates.

This might be less of a concern for Realm VMs. I'm wondering whenever it is actually possible to support cpu hotplug (or memory hotplug) for Realm VMs, and there are probably more features which are not useful for CC guests.

So the idea to have a single Realm VM firmware image and have the image generate ACPI tables from FDT looks workable to me.

[kraxel]

While being at it: Setting PcdConfidentialComputingGuestAttr probably makes sense.

[lgao4]

The change in MdePkg is OK to me.