Complete composefs integration in Fedora CoreOS

[travier]

Describe the enhancement

We should complete the integration of composefs in Fedora CoreOS. Composefs brings better security and potentially better performance (no need to create a deploy anymore).

Support for composefs already partially landed in ostree and coreos-assembler. Now we need to figure out the missing pieces, the integration and testing.

System details

All

Additional information

See:

• https://github.com/containers/composefs
• https://blogs.gnome.org/alexl/2023/07/11/composefs-state-of-the-union/
• https://blogs.gnome.org/alexl/2024/01/15/testing-composefs-in-silverblue/
• Strategy for file verification (IMA, fs-verity, composefs) #1252
• ex-integrity.composefs: Tracking issue ostreedev/ostree#2867

[jlebon]

Also see coreos/fedora-coreos-config#2856. I think essentially we need to pick up that PR and push it through to get it running in rawhide at least. But note coreos/fedora-coreos-config#2856 (comment):

A thing that this is known to break is the "chattr -i" hack for new toplevel dirs (xref coreos/rpm-ostree#337 )

What I don't know offhand is how problematic that will be in reality. My instinct says we can do this, but with a sufficiently loud announcement.

But note in the context of bootable containers, you can create whatever top-level directory you'd like at derivation time. The tricky bit is if you need to create dynamically named top-level directories. In that case... those use cases might need a tmpfs overlay on top (e.g. the rootfs.transient work, though ideally you could create the overlay, add your directories and then change the overlay back to read-only, but that'd need to be done from the initramfs I think).

[travier]

Some edited notes from the meeting:

Composefs is a combination of filesystems (overlayfs, erofs) that creates an ostree like system without some of the downsides of ostree.

It also provides runtime integrity checks where ostree "only" provides offline integrity checks.

Combined with podman / container storage support, you can even gain disk space and memory de-duplication if your containers use the same binaries as your system. If we share the ostree/composefs repo with the podman system container store then we can de-dup the files with the same hash.

It also has potential (small) performance improvements for deployments/updates as you don't need to create an entire filesystem tree for /usr anymore for each version as those are fully stored as EROFS images. Ostree creates a hardlink farm for each deployment, but you can not hardlink folders and symlinks so you need to create the entire tree. With composefs, the filesystem hierarchy (mostly paths and metadata) is stored in the EROFS filesystem, which is a read only FS stored in a file. The file content remain in the ostree repo / composefs store. Compoefs uses overlayfs to reference the files stored in the ostree repo, via the filesystem stored as EROFS.

In the first phase AIUI, ostree will still create deployments anyway, but we would dynamically create the composefs image at boot.

[jbtrystram]

Writing up some investigation I did today around signing the ostree commit with a key that we embed into initramfs to leverage composeFS validation.

We need to sign the ostree commit with an Ed25519 key, likely during the Build OSTree stage of the pipeline. At the moment i don't think we do that, looking at coreos-assembler/src/cmd-build.
Then we can embed the pubkey in initramfs and wire up things.

I initially thought that would interfere with the robosignatory step but after writing this up I don't think so. Still worth asking :)

https://ostreedev.github.io/ostree/composefs/#signatures

[travier]

ostreedev/ostree#3256

[jbtrystram]

initial experiment : coreos/coreos-assembler#3813

[travier]

Draft change in https://fedoraproject.org/wiki/Changes/ComposefsAtomicCoreOSIoT

[jlebon]

We discussed this in today's community meeting. One thing that came up was about how to roll this out to existing nodes.

Currently, as it's enabled via a /usr dropin, it would affect existing nodes and new installs simultaneously. If those could be decoupled, then we could e.g. let it bake even in stable for a month or two on new installs before enabling it for upgrading nodes.

Regardless of the approach, once this hits next, this should be part of our communications to raise awareness and invite testing.

[jlebon]

xref: ostreedev/ostree#3277

[travier]

This has been approved by FESCo: https://pagure.io/fesco/issue/3240

[jbtrystram]

this is currently stuck on https://bugzilla.redhat.com/show_bug.cgi?id=2284097

[jbtrystram]

The kdump issue should be resolved with rhkdump/kdump-utils#28

[travier]

Documentation needed:

• How to disable it (karg) (https://gitlab.com/fedora/bootc/tracker/-/issues/27)
• Top-level directories (https://gitlab.com/fedora/bootc/tracker/-/issues/26)
• kdump (https://gitlab.com/fedora/bootc/tracker/-/issues/19)

So it's probably best to make a dedicated page that would start by explaining quickly what composefs is about and would then discuss the details above.

We should also give a heads to the podman team and help them set things up to disable it for them.

[jlebon]

We should also give a heads to the podman team and help them set things up to disable it for them.

cc @baude @mheon

[travier]

Documentation: https://docs.fedoraproject.org/en-US/fedora-coreos/composefs/

[travier]

The fix for this went into next stream release 41.20240916.1.0. Please try out the new release and report issues.

[dustymabe]

The fix for this went into testing stream release 41.20241027.2.0. Please try out the new release and report issues.

[dustymabe]

The fix for this went into stable stream release 41.20241027.3.0.