RPM-OSTree - git for operating system binaries

[schaefi]

With rpm-ostree, you can create versioned filesystem trees to provide atomic, reliable upgrades. "git-like" management of appliances would be pretty sweet.

[schaefi]

https://wiki.gnome.org/Projects/OSTree

[Conan-Kudo]

RPM-OSTree package for openSUSE Leap 15.0 / Tumbleweed: https://build.opensuse.org/package/show/home:Pharaoh_Atem:SUSE_RPMOSTree/rpm-ostree

Docs and examples:

• https://github.com/projectatomic/rpm-ostree/blob/master/docs/manual/treefile.md
• https://pagure.io/fedora-atomic/tree/master
• https://pagure.io/workstation-ostree-config/tree/master
• https://pagure.io/fork/siosm/workstation-ostree-config/tree/f28-kinoite

[davidcassany]

@Conan-Kudo I am still wondering two things:

1. What actually means supporting rpm-ostree in KIWI? Probably this caused by my lack of knowledge regarding rpm-ostree. I assume this is a new package manager right? or at most an extension of the feature set of DNF and/or any other that has some layer of cooperation.

2. Which is the use case? According to what I understand this is not much different of building a root-tree over an ostree, thus building rootfs in some container like layered filesystem, correct? So I am wondering what is actually the result of using rpm-ostree as a package manager for an e.g. Fedora build. Mostly because if rpm-ostree is a package manager (from KIWI POV) then it should not have impact on the build procedure. It should still be possible to build isos, vms, oem, etc.

I can imagine features like usign the derived_from attribute to import an ostree and create a new images out of it by adding a new transaction on top with rpm-ostree. Am I pointing to the right direction?

Also stuff like making each <packages> section as an individual transaction, even I am having hard time to see the use case and potential benefits of it.

We need to refine this support to narrow what actually means and also understand the use cases we are covering with it. @Conan-Kudo would be appreciated if you can help us on that regard.

[Conan-Kudo]

RPM-OSTree produces an OSTree, which is a special rootfs type representing a largely immutable operating system environment. The rpm-ostree program has features for producing and managing OSTrees from RPM content. It does leverage DNF stack libraries, but it does not have the same interface as DNF.

What it would mean from a KIWI perspective is that it would be able to orchestrate creation of OSTrees and applying them to the desired medium so that they can be used in the same way traditional Linux system trees are.

Today, there aren't any tools that make it easy to produce RPM-OSTree based things other than VM disk images with cosa. As KIWI can produce way more types and is easier to leverage (think OBS and similar), that would give it a major leg up for producing immutable infrastructure OSes. As it is, I currently cannot create a live ISO of an rpm-ostree based system, nor can I easily create layered OSTree repositories for feeding update repos or derivative distros.

@cgwalters and @jlebon would have a better idea of the exact mechanics here, as they're the current developers of rpm-ostree.

[cgwalters]

I currently cannot create a live ISO of an rpm-ostree based system,

cosa buildextend-live is used to generate the "live ISO" linked from the FCOS builds.

[schaefi]

Another player in the field of "packages" is luet

• https://luet-lab.github.io/docs/docs/

the term package in this regard means a container+plus_metadata

the term package in the context of rpm-ostree is an extended format for rpm (iirc)

In any case kiwi has no support for the sophisticated new idioms of packages which are actually sub-systems.
As of now I still don't know if there is a high demand to natively support these type of package managers in kiwi.

kiwi itself is flexible enough to use any os-tree you will give it. Therefore it's possible to combine rpm-ostree
or luet or whatever else will come providing "systems" to let those tools create/manage a new root tree and let
kiwi take over only for the task to create images out of it

From the perspective of a stacked solution I currently don't see the need to add work into kiwi.

However it might be a good idea to discuss a change in kiwi that makes the "installation of packages" as we see
it now an optional step. Currently the following is possible with kiwi:

$ kiwi-ng system build --description ... --target-dir ... --allow-existing-root

This will re-use an eventually existing root tree no matter who created it, luet, rpm-ostree, etc...
But the provided description still needs:

• at least one old school package repository from the distribution
• at least on <packages> section even if you don't install anything in addition to the existing tree

This tells us that kiwi itself expects some connection to repositories and packages as they are provided by the distribution vendors. The discussion we can have here is if we want to weaken this concept if customers can provide a system root tree to work with ?

Your feedback is welcome

Thanks

[davidcassany]

However it might be a good idea to discuss a change in kiwi that makes the "installation of packages" as we see
it now an optional step. Currently the following is possible with kiwi:

$ kiwi-ng system build --description ... --target-dir ... --allow-existing-root

This will re-use an eventually existing root tree no matter who created it, luet, rpm-ostree, etc...
But the provided description still needs:

* at least one old school package repository from the distribution

* at least on `<packages>` section even if you don't install anything in addition to the existing tree

This tells us that kiwi itself expects some connection to repositories and packages as they are provided by the distribution vendors.

I have thinking a quite a lot within the last few days regarding these topics. First things regarding the demand that there might be regarding these sort of ostree/luet/container like packages do think there is no demand on adding support at package manager level. It is certainly possible (I already did a successful PoC for Luet as a hobby thing) but even I believe it might make sense in some specific use cases I also think this is not best approach to take advantatge of these sort of technologies from an appliance building PoV.

What is appealing from containers? IMHO from developers PoV, beyond the isolation and the DevOps trends, something that is nice is the stacking or layering concept. You never start from scratch and you can always add some little salt to anything and turn something that was close to what you wanted to something else that really matches your needs or expectations. With all that you probably don't really need to care much about how your base layer was created, what is actually including and all configuration details. Damn easy to get started.

What if we make use of container technologies to build the idea of derived images for any type of image. Imagine that after the prepare step the created root tree (including all KIWI artifacts) is kept in some ostree or OCI storage/registry. So we could eventually rebuild the exact same image by just pulling in this stored prepare step (it would already contain the XML description and any other artifact that KIWI requires to create an image). Alternatively, we could recreate the whole tree (re run the prepare step on top of the already provided tree so, for instance, it could upgrade packages and rebuild again the provided image). But what if we also provide a delta of a new XML description? What if in that procedure we allow to include XML description artifacts that are simply merged with the base XML, so on the fly a new full XML gets created and applied on top of the imported rootfs.

Imagine we have a Live image, with a <type> like:

<type image="iso" flags="overlay" firmware="efi" kernelcmdline="splash" hybridpersistent_filesystem="ext4" hybridpersistent="true">
   <rootfs-base keep="oci:myliveimage:latest"/>
</type>

With all the packages and so one. So the prepare step of this image could eventually push the root-tree into an OCI image myliveimage:latest and the end of it. So a kiwi-ng system build call, as part of the build process, stores the WORKING root-tree in some OCI storage for any potential later reuse.

Then imagine I want to tweak this image and build a new one on top of this one. Imagine we could do something like:

<image>
<derived from="myliveimage:latest"/>
<packages type="image">
   <package name="my-new-branding"/>
</packages>
<packages type="uninstall">
   <package name="my-old-branding">
</packages>
</image>

And a kiwi-ng system build call on that system could eventually result into the former base XML with these additional packages sections appended and executed the build over the already provided root-tree as it happens with --allow-existing-root. Taking it one step further this could even be used to build a different type of image or build the same image but using dmsquash instead of overlay... The obvious problem here is how to define the XML merge and some details of the build steps (create only vs prepare and create). So some questions to figure out, but I believe it should be possible to find some simple and comprehensible procedure for those.

The discussion we can have here is if we want to weaken this concept if customers can provide a system root tree to work with ?

So, what if there is no need to weaken this concept, what if we enforce the imported rootfs to be a fully qualified KIWI rootfs, including the XML, config.sh, etc.

Something like that provides 100 ways to shoot yourself, of course, but you can already shoot yourself quite easily at the moment if you start using all the schema possibilities randomly.

This is probably a dumb idea, but I believe there should be a way to build images on top of another image and at the moment this is the only "sane" procedure that came into my mind :P

Any thoughts?

[cgwalters]

If it helps, iit's somewhere on the roadmap for me to better integrate (rpm-)ostree with container images, and make the experience support something much more like what you're talking about - a Dockerfile-style FROM fedoracoreos then do stuff. A lot of stuff involved there though.

[schaefi]

I like the idea to preserve a kiwi root tree in a registry. Correct me when wrong, the stored root tree would be a docker container and can also be used as such but its main use case is to preserve a root tree from a kiwi stage for further operations as you mentioned, right ?

I think this is a great concept to implement the "reproducable build feature" you had in mind earlier and allows for more use cases.

From a docker registry perspective the pushed data is probably not very useful as a container. Thus I was thinking if providing the tree as an ostree would fit in better. However I must say that using a docker registry and the handling of data as a docker container is pretty straight forward and easy compared to an rpm ostree. But this could also because of my lack of knowledge on the ostree end

But what if we also provide a delta of a new XML description?

I had played with this in the past and I must say it was very hard to achieve this on the XML level. A generic approach that allows kind of a XML merge is possible but ends in a cumbersome to use concept. The delta part will not validate the schema only the combined information will be valid. The model I came up then as an alternative was the profiled sections, but this doesn't play well on a delta concept. So the idea is nice but for a delta concept I suggest to use a meta language on top which can be composed into a complete XML description for validation. Similar to the C prepocessing

[davidcassany]

Closing the issue since we are definitely deviating from the original topic and no progress is expected. Also related ideas are discussed in #1771

[Conan-Kudo]

I was talking to @travier today in the Fedora KDE SIG meeting, and he said he'd be interested in helping us understand how to add RPM-OSTree support to KIWI.

[dcermak]

So, I have been playing around a teeny-tiny bit with ostree and unfortunately I am still far from really understanding it, but this is roughly how I got it:

1. ostree itself will build a ostree commit, which is effectively just a FS snapshot that ostree can grab and deploy on an existing system
2. you setup a http server and serve this ostree commit to your running system, point ostree to that webserver and run a ostree upgrade
3. ostree will apply that commit and make it available after a reboot

What I have not found out yet, is how you create a disk/iso image from this.

However, an ostree commit is afaik just a tarball of the filesystem, so if we can create a VM/iso from a tarball, then we should be able to build a system with ostree enabled. But admittedly, this is all very hand-wavy at the moment.

[Conan-Kudo]

This code from osbuild shows how they generate an rpm-ostree commit, and that might be useful for coming up with a way to do it in kiwi.

[cgwalters]

Since openSUSE now uses /usr/etc

Link/more info?

[Conan-Kudo]

Since openSUSE now uses /usr/etc

Link/more info?

openSUSE now uses /usr/etc to store vendor configs of stuff that is in /etc, and my understanding of the situation is that this is the location RPM-OSTree uses to divert and archive pristine copies of /etc. At least, I think that's what you told me when you removed /usr/etc from Fedora's filesystem package years ago.

Since RPM-OSTree does not use a private hierarchy for this, it effectively has become incompatible with openSUSE (not that there weren't already other issues with its usage of %verifyscript and a few other things, but this is the final nail in the coffin).

[travier]

I've been reading on how kiwi works and I think that we should do the following:

• Add a prepare step that takes a ostree native container image and deploys it to a sysroot folder.
  • https://osinside.github.io/kiwi/concept_and_workflow.html#the-prepare-step
  • It will be an "alternative" to the package-based options that are in kiwi right now
• Then figure out if the existing create step works for the image types that we care about:
  • https://osinside.github.io/kiwi/image_types_and_results.html
  • Ideally for ISO Hybrid Live Image and Virtual Disk Image

We will add support for building ostree native containers to kiwi later. We will also need to figure out where bootupd fits in there but we can do that later as well.

Let me know what you think.

[Conan-Kudo]

This seems like a reasonable plan. So then the end state we should be able to create live media and disk images that are RPM-OSTree backed?

[schaefi]

I was wondering if the kiwi stackbuild plugin could already be helpful here. See the following documentation

• https://osinside.github.io/kiwi/plugins/stackbuild.html#turn-a-container-into-a-vm-image

Let's say I take this and replace the example container with an ostree native container I should be able to produce an RPM-OSTree backed image. I would be eager to try that out. Where can I find ostree containers ?

Thanks

[travier]

So then the end state we should be able to create live media and disk images that are RPM-OSTree backed?

Ideally yes.

I was wondering if the kiwi stackbuild plugin could already be helpful here

I don't think this will work. Ostree needs special handling to deploy a version to a system rootfs.

Overall, we'll have to duplicate the logic from https://github.com/coreos/coreos-assembler/blob/main/src/create_disk.sh#L312..L379 (https://manpages.debian.org/testing/ostree/ostree-admin-deploy.1.en.html).

[travier]

I would be eager to try that out. Where can I find ostree containers ?

https://quay.io/repository/fedora-ostree-desktops/kinoite?tab=tags

[schaefi]

I don't think this will work. Ostree needs special handling to deploy a version to a system rootfs.

yeah I see and that matches my results. So here is what I did

• install the kiwi-stackbuild-plugin from pip or the package as documented here: https://osinside.github.io/kiwi/plugins/stackbuild.html
• mkdir ostree-to-live
• Add the relevant image description to build a live image from an ostree container, vi ostree-to-live/config.kiwi

<?xml version="1.0" encoding="utf-8"?>

<image schemaversion="8.0" name="Rawhide-Live">
    <description type="system">
        <author>The Author</author>
        <contact>[email protected]</contact>
        <specification>
            Live Image Build OS-tree backed using rawhide
        </specification>
    </description>
    <preferences>
        <type image="iso" flags="dmsquash" firmware="uefi" hybridpersistent_filesystem="ext4" hybridpersistent="true"/>
        <version>1.42.1</version>
        <packagemanager>dnf4</packagemanager>
        <locale>en_US</locale>
        <keytable>us</keytable>
        <timezone>UTC</timezone>
    </preferences>
    <repository type="rpm-md" alias="Rawhide" sourcetype="metalink">
        <source path="https://mirrors.fedoraproject.org/metalink?repo=fedora-rawhide&amp;arch=x86_64"/>
    </repository>
    <packages type="bootstrap">
        <package name="grub2"/>
        <package name="grubby"/>
        <package name="kernel"/>
        <package name="grub2-efi-x64-cdboot"/>
        <package name="dracut-live"/>
    </packages>
    <users>
        <user password="$1$wYJUgpM5$RXMMeASDc035eX.NbYWFl0" home="/root" name="root" groups="root"/>
    </users>
</image>

• build the image selecting an ostree rawhide container

sudo kiwi-ng system stackbuild --stash kinoite:rawhide --from-registry quay.io/fedora-ostree-desktops --target-dir /tmp/my-ostree-live --description ostree_to_live

This works and builds a live image, but it's huge :)

I was looking into the rootfs and I found the specials when using ostree.

total 24
dr-xr-xr-x.   2 root root    6 Mar 13 01:00 afs
lrwxrwxrwx.   1 root root    7 Mar 13 01:00 bin -> usr/bin
dr-xr-xr-x.   5 root root 4096 May 14 17:47 boot
drwxr-xr-x    3 root root   92 May 14 17:49 dev
drwxr-xr-x. 136 root root 8192 May 14 17:49 etc
drwxr-xr-x.   2 root root    6 Mar 13 01:00 home
drwxr-xr-x.   3 root root   38 May 14 17:49 image
lrwxrwxrwx.   1 root root    7 Mar 13 01:00 lib -> usr/lib
lrwxrwxrwx.   1 root root    9 Mar 13 01:00 lib64 -> usr/lib64
drwxr-xr-x.   2 root root    6 Mar 13 01:00 media
drwxr-xr-x.   2 root root    6 Mar 13 01:00 mnt
drwxr-xr-x.   2 root root    6 Mar 13 01:00 opt
lrwxrwxrwx.   2 root root   14 Jan  1  1970 ostree -> sysroot/ostree
drwxr-xr-x    2 root root    6 May 14 17:46 proc
dr-xr-x---.   3 root root   18 May 14 17:47 root
drwxr-xr-x.  34 root root 4096 May 14 17:47 run
lrwxrwxrwx.   1 root root    8 Mar 13 01:00 sbin -> usr/sbin
drwxr-xr-x.   2 root root    6 Mar 13 01:00 srv
drwxr-xr-x    2 root root    6 May 14 17:46 sys
drwxr-xr-x.   3 root root   20 Jan  1  1970 sysroot
drwxrwxrwt.   2 root root    6 May 14 17:49 tmp
drwxr-xr-x.  12 root root  144 May 14 17:47 usr
drwxr-xr-x.  23 root root 4096 May 14 17:47 var

in /sysroot/ostree I could see all the different objects that makes up that ostree.

At this point we need to add the deploy functionality that you pointed me to, but thanks for sharing the containers and the link, I understood a little bit more on the concept :)

[travier]

So far I can do most things using scripts and not touching kiwi code. WIP PR with the descriptions in https://pagure.io/fedora-kiwi-descriptions/pull-request/58.

This is currently failing on:

[ ERROR   ]: 10:39:39 | KiwiKernelLookupError: No kernel found in /home/fedora/fedora-kiwi-descriptions/outdir/build/image-root, searched for []

as ostree does not install the kernel in /boot but in /boot/ostree/fedora-<hash>/...:

$ tree outdir/build/image-root/boot/ostree
[drwxr-xr-x root     root       142]  outdir/build/image-root/boot/ostree
└── [drwxr-xr-x root     root       204]  fedora-738613c8ada41ad14ff1b4bec3e06ab89c3948aed6e614844730e23de50ea337
    ├── [-rw-r--r-- root     root      131M]  initramfs-6.8.10-300.fc40.x86_64.img
    └── [-rwxr-xr-x root     root       15M]  vmlinuz-6.8.10-300.fc40.x86_64

Does it needs to find the kernel to generate the GRUB/bootloader config? In our case, we use the BLS configs so we don't need that.

It's likely that we'll also install the bootloader using bootupd soon.

[Conan-Kudo]

Does it needs to find the kernel to generate the GRUB/bootloader config? In our case, we use the BLS configs so we don't need that.

It does if we want to be able to build ISOs.

[schaefi]

We need to be able to identify a kernel. In the container world the kernel is shared with the host and you simply don't care but if you want to boot a system no matter if it's an ISO or a disk image we need to be able to make a choice for the kernel and also find the kernel binary.

In kiwi the version lookup is based on the versioned directory name of the kernel in either /lib/modules or /usr/lib/modules and so far we trusted the sub-directories below these paths to name a kernel version e.g
/lib/modules/5.3.18-59.10-default and the kernel version is then 5.3.18-59.10-default

In case of an ostree I don't understand how it's done. The ostree can provide a number of objects, e.g
/boot/ostree/fedora-738613c8ada41ad14ff1b4bec3e06ab89c3948aed6e614844730e23de50ea337/vmlinuz-6.8.10-300.fc40.x86_64. To match a kernel version the name of the kernel file is the only source that gives a hint.
But how does this match the kernel version when I can have a thousand objects ?

[cgwalters]

For ostree and bootc systems, the kernel is canonically in /usr/lib/modules/$kver - the /boot/ostree is an implementation detail that other projects should not peek into, basically.

Alternatively of course, you can look at the bootloader entries in /boot/loader and what those point to.

[travier]

As far as I can tell, it looks like this is exactly the logic in:

• kiwi/kiwi/system/kernel.py

  Line 58 in ce10880

  def get_kernel(

• kiwi/kiwi/system/kernel.py

  Line 157 in ce10880

  def _setup_kernel_names_for_lookup(self) -> List[str]:

but if I understand correctly, it did not look for them in the ostree deployment, but in the "sysroot", where we only have the copy in /boot/ostree so it did not find the kernels.

But ideally kiwi does not have to find the kernel as it should let the bootloader follow the BLS config that has been written by ostree.

I'll make a temporary workaround and then we can circle back to this once I understand better how kiwi works.

[travier]

With #2557 and https://pagure.io/fedora-kiwi-descriptions/pull-request/58, I get a booting QCOW2 image!

[travier]

I made a few additional hacks and got a LiveISO working!

[travier]

The next step I need help with for the Live ISO is figuring out a way to read a BLS entry from the image dir to get the ostree kernel command line argument and "pass" them to the final kernel command line for the Live ISO GRUB config.

Another option is to read those kernel command line arguments from the image dir in the GRUB config step but so far I haven't been able to find a way to do that. Maybe I missed the right variable with the path to the root dir.

How do we pass information in general from the prepare step to the image step?

[schaefi]

The next step I need help with for the Live ISO is figuring out a way to read a BLS entry from the image dir to get the ostree kernel command line argument and "pass" them to the final kernel command line for the Live ISO GRUB config.

There is a kiwi hook script called editbootconfig which allows to do custom changes to the bootloader configuration. Also see: https://osinside.github.io/kiwi/image_description/elements.html. Be careful when using this script hook because it does not run chrooted

There is also the kernelcmdline attribute in the <type> section of the image description which allows to add custom kernel boot options to the image. However this is only useful for static data that you would like to pass to the kernel, dynamic information such as UUID's or similar doesn't make sense to be placed there

How do we pass information in general from the prepare step to the image step?

The job of the prepare step is to create the data of the image root-tree. Any information that is needed to let the create step pass is expected to be present inside of the root tree such that the create step can do the job. Sometimes people create temporary (present during build time) data and delete them later by one of the available script hooks. Usually this sort of tricks are only needed for bootloader related specialities which requires to be done late in the processing.

In this particular case for building a live ISO, kiwi uses its own grub template which you can overwrite with your own template file. See https://osinside.github.io/kiwi/image_description/elements.html and search for grub_template

Hope this helps

[ericcurtin]

The next step I need help with for the Live ISO is figuring out a way to read a BLS entry from the image dir to get the ostree kernel command line argument and "pass" them to the final kernel command line for the Live ISO GRUB config.

Another option is to read those kernel command line arguments from the image dir in the GRUB config step but so far I haven't been able to find a way to do that. Maybe I missed the right variable with the path to the root dir.

How do we pass information in general from the prepare step to the image step?

In osbuild, the scripts assume one version of the OS in installed in the rootfs, so based on that assumption they just populate the grub bls file with values that make sense from querying the filesystem. So we can make the same assumptions. If we have two kernels initially or example rpm-ostree doesn't like it anyway.

And when we do need multiple versions of the OS, it's easier just to do multiple builds.

[ericcurtin]

I am thinking of the Asahi case more than the .iso case though which is likely missing the point.

[travier]

However this is only useful for static data that you would like to pass to the kernel, dynamic information such as UUID's or similar doesn't make sense to be placed there

That's the case here. I don't know the ostree<hash> command line parameter in advance and I need to read it from the BLS config that ostree generated to add it to the GRUB config used for the ISO.

[travier]

So, to make this more integrated, I'm looking at creating a new "package manager" that would take a special "package" for the "image" type/phase that would be the container image URL. I still need the bootstrap package manager & packages to get the tools in the bootstrap env to install this container.

Not sure if that makes sense / how to integrate that properly?

[Conan-Kudo]

Yes, that makes sense. You'd probably want to make the bootstrap package manager just be a stub that uses the DNF class, and then for the image section, you'd use rpm-ostree that was installed in bootstrap.

[securitybrahh]

Hope to see Asahi AAA gaming with disk encrpted ostree, one day!

Thanks for making the best machine ever m1n1.