Publish meta-data for use with virt-manager

[brianredbeard]

To support the existing libvirt tooling used across Red Hat products we should support the instantiation of hosts with virt-install. To do this most effectively we will need to coordinate the following:

• Publish .treeinfo metadata which can be consumed by virt-manager based tooling.
• Utilize said metadata so that it can select the correct binaries for virtualized hardware setup (especially as we work through various hardware enablement like enabling ppc64le for Fedora CoreOS #78 (enabling ppc64le for Fedora CoreOS), Support userdata on QEMU s390x ignition#825 (Support userdata on QEMU s390x), and Support userdata on QEMU PPC64 ignition#666 (Support userdata on QEMU PPC64)
• Ensure Fedora CoreOS is represented correctly in osinfo-db -> https://gitlab.com/libosinfo/osinfo-db/-/issues/68
• Work with the virt-manager project to ensure any needed primitives for Ignition are handled.

[brianredbeard]

A corresponding issue has been filed for virt-manager in https://bugzilla.redhat.com/show_bug.cgi?id=1731219

[mscherer]

So, if we could have the .treeinfo, that would already be useful, cf https://arrfab.net/posts/2020/Sep/11/deploying-openshift-in-kvmlibvirt-guests/

[lucab]

I'm having some basic doubts about this ticket, and I'd be happy to have some treeinfo/virt-manager expert chime in here to help clearing those.

All the treeinfo refs/examples I've seen so far are describing a set of ISOs, or kernel+fs artifacts. However for FCOS we do publish qcow2 images for the qemu platform, which are the preferred media in this case and the ones mentioned by @brianredbeard. How would that work from the point of view of treefile and virt-manager?

[crobinso]

AFAICT treeinfo would only be relevant for the installer iso. Maybe FCOS doesn't publish one, but RHCOS does, for example:

https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.5/latest/rhcos-installer.x86_64.iso

Inside there's an images/ dir with kernel/initrd in it. The layout is slightly different than standard Fedora/RHEL install trees, example: https://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/Server/x86_64/os/

That latter URL has a .treeinfo file which applications can parse to find for example the architecture of the kernel in the tree, and the location of the kernel+initrd in the tree. Also some metadata that apps can use to identify the media in isolation (Name: Fedora Rawhide for example)

virt-install/virt-manager can boot off that media as a CDROM without needing to know kernel/initrd location. But if the user wants to pass in kernel arguments, qemu needs to boot the kernel directly, meaning virt-install needs to extract the kernel from the ISO, meaning it needs to know where the kernel is on the iso. That's where .treeinfo comes in. FWIW all other RH based distros (and suse nowadays) put a treeinfo file on their non-live install media and in their URL trees

[cgwalters]

But if the user wants to pass in kernel arguments, qemu needs to boot the kernel directly

We are actually working on support for injecting kernel arguments into the ISO, see coreos/coreos-assembler#1637

FWIW all other RH based distros (and suse nowadays) put a treeinfo file on their non-live install media and in their URL trees

Fedora CoreOS only has a Live ISO that can also be used for installation - there's no special installer ISO. RHCOS after OpenShift 4.6 and above will also match this, see https://github.com/openshift/enhancements/blob/master/enhancements/rhcos/liveisoinstall.md

[jlebon]

We did work recently to put the PXE artifacts in the ISO. They're at the standard /images/pxeboot directory.

But if the user wants to pass in kernel arguments, qemu needs to boot the kernel directly, meaning virt-install needs to extract the kernel from the ISO, meaning it needs to know where the kernel is on the iso.

I started hacking on adding a .treeinfo to the live ISO though note also that one thing which complicates matters is that the rootfs is in a separate initrd (also in /images/pxeboot). And since it doesn't seem like qemu supports multiple initrds, users would still have to e.g. pass the rootfs via -cmdline coreos.live.rootfs_url=http://... (see https://docs.fedoraproject.org/en-US/fedora-coreos/bare-metal/#_pxe_rootfs_image). What would be nice is if virt-install/virt-manager could learn to concatenate the two initrds in a tmpfile and pass that as the initrd to qemu.

Edit: threw up coreos/coreos-assembler#1733.

[bgilbert]

Assuming the ISO image still gets attached to the VM, it would also work for the .treeinfo file to carry the coreos.liveiso argument that tells the initrd to mount the rootfs directly from the ISO. But the document linked above doesn't specify any way for the .treeinfo to specify default kernel arguments. How does this normally work? Does a .treeinfo direct kernel boot really remove all OS control of the resulting kargs?

[crobinso]

Pulling out kernel/initrd from the ISO is typically used with full install tree DVDs and netboot isos that boot straight into the distro installer instance, where it's useful for users to have access to kernel args so they can pass in things like kickstart files.

If the ISO is intended as live media, then IMO it's expected that virt-install does not know how to extract kernel/initrd from that ISO. We skip that for other distro live media too.

FWIW my initial treeinfo suggestion came from https://bugzilla.redhat.com/show_bug.cgi?id=1731219 which was about use of RHEL CoreOS media

[jlebon]

Pulling out kernel/initrd from the ISO is typically used with full install tree DVDs and netboot isos that boot straight into the distro installer instance, where it's useful for users to have access to kernel args so they can pass in things like kickstart files.

If the ISO is intended as live media, then IMO it's expected that virt-install does not know how to extract kernel/initrd from that ISO. We skip that for other distro live media too.

Our live ISO plays double duty. It also works like a typical install ISO that can be configured solely from kernel args. As Colin said, we're working on adding kargs support for the ISO, though it'd be nice if virt-install Just Worked too with FCOS. (And eventually, I think it'd be nice to support the "remote tree" case as well.)

In coreos/coreos-assembler#1733, I add a new rootfs key under the [images-$arch] group. WDYT about virt-install knowing about this and appending the two together? (Or we could keep it generic and call it e.g. initrd2). That would help with both cases.

[bgilbert]

@jlebon That'd work, but I think adding treeinfo support for default kernel arguments would be more generally useful outside of FCOS/RHCOS. Then we could mount the rootfs from the ISO and keep it out of RAM entirely.

[jlebon]

@jlebon That'd work, but I think adding treeinfo support for default kernel arguments would be more generally useful outside of FCOS/RHCOS. Then we could mount the rootfs from the ISO and keep it out of RAM entirely.

Right, agreed. I was trying to address both cases (ISO and remote tree) in one RFE. Definitely an improvement in the ISO case if we can pull the rootfs directly from there.

[crobinso]

@jlebon I would like to avoid merging the two initrds. In this area virt-install is exclusively concerned with locating kernel/initrd location in a tree, I would rather not open the door to performing default media manipulations for distros, sounds like a slippery slope TBH

I know CoreOS is different in a lot of ways so I'm not sure what is relevant here but let me provide more info about how other distros work WRT virt-install.

Say a user does virt-install --location https://dl.fedoraproject.org/pub/fedora/linux/releases/32/Server/x86_64/os/ .... virt-install/libosinfo downloads $URL/.treeinfo, finds paths for kernel+initrd from that, downloads them both, passes them to qemu. We also pass inst.repo=$URL on the kernel command line. We get inst.repo from the <kernel-url-argument> for fedora32 in osinfo-db. anaconda first stage in the initrd we downloaded knows to walk that URL to pull down stage2 media from the proper path and continue with the install. We do something similar for opensuse too.

For the ISO install case, if requested to extract kernel/initrd so user can pass in custom args, virt-install doesn't add any kernel arguments ourselves. AFAIK for all distros that virt-install supports, when booting directly from kernel+initrd and there is CDROM attached install media, it does the right thing by using the local installer (if it's DVD) or grabbing an install URL from a known location (if it's netboot).

You mention there is the kernel arg coreos.live.rootfs_url=https:// which kinda sounds similarish to inst.repo, which is why I'm mentioning these cases, maybe these patterns can converge somehow

[bgilbert]

Thanks, those details are helpful.

We get inst.repo from the <kernel-url-argument> for fedora32 in osinfo-db.

This comment initially confused me, so to clarify: the name of the inst.repo argument, not the value, is what comes from osinfo-db.

anaconda first stage in the initrd we downloaded knows to walk that URL to pull down stage2 media from the proper path and continue with the install. We do something similar for opensuse too.

We could extend coreos-livepxe-rootfs.sh to read the inst.repo argument, fetch a treeinfo file, and read the rootfs filename out of that. The larger issue with this flow, I think, is the notion that there's a well-known directory on a webserver that the user can paste into --location. As it happens, we group the artifacts for one release into a directory in the builds bucket. But we don't promise to maintain that structure, since users should be getting individual artifact URLs from stream metadata. And the directory name for the latest release will change every two weeks at least.

#625 is relevant here; maybe the well-known URL should be a redirector that reads stream metadata.

AFAIK for all distros that virt-install supports, when booting directly from kernel+initrd and there is CDROM attached install media, it does the right thing by using the local installer (if it's DVD) or grabbing an install URL from a known location (if it's netboot).

How does the distro know whether to wait for local media or to netboot? When virt-install does a direct kernel+initrd boot, does it pass different kernel arguments if there is or isn't a local ISO?

---

So I'm currently seeing three high-level issues:

1. The preferred way to install FCOS on QEMU/KVM is to download the FCOS QCOW artifact, but it appears that virt-install doesn't support that natively. We can probably document our way out of that with coreos-installer download + virt-install --import --disk, but that's a bit messier.
2. virt-install direct kernel booting doesn't allow the OS to specify default kernel arguments. We can partially work around that with inst.repo, possibly at the cost of redownloading the rootfs. But we'd fail to set the installed platform ID correctly unless the user set it, or e.g. inst.repo engaged an ugly platform-detection hack.
3. To ergonomically support .treeinfo on our download server, we'd probably need a redirector service. That's suboptimal but possible.

Would the first two be good candidates for virt-install RFEs?

On the upside, --initrd-inject with direct kernel booting provides a nice way to support Ignition configs without thinking about -fw_cfg, provided that the user names their config exactly config.ign. (--qemu-commandline is supported too, so -fw_cfg should also work.)

[crobinso]

This comment initially confused me, so to clarify: the name of the inst.repo argument, not the value, is what comes from osinfo-db.

That's correct, sorry for not being more clear.

We could extend coreos-livepxe-rootfs.sh to read the inst.repo argument,

One note: I wasn't implying inst.repo naming needs to be preserved, that's an anaconda detail. Feel free to name it whatever makes most sense, like with a 'coreos' prefix

AFAIK for all distros that virt-install supports, when booting directly from kernel+initrd and there is CDROM attached install media, it does the right thing by using the local installer (if it's DVD) or grabbing an install URL from a known location (if it's netboot).

How does the distro know whether to wait for local media or to netboot?

I don't know. Maybe @bcl can help. Brian can you check the two previous comments and correct our understanding here.

When virt-install does a direct kernel+initrd boot, does it pass different kernel arguments if there is or isn't a local ISO?

We only pass default kernel arguments (inst.repo etc) if we are installing from a network tree. All other cases including local iso we don't pass any arguments. Somehow it 'just works' :)

So I'm currently seeing three high-level issues:

The preferred way to install FCOS on QEMU/KVM is to download the FCOS QCOW artifact, but it appears that virt-install doesn't support that natively. We can probably document our way out of that with coreos-installer download + virt-install --import --disk, but that's a bit messier.

If we had Fedora CoreOS entries in osinfo-db, which point to stable locations for qcow2 disk images, we could extend virt-install to pull these down automatically virt-install --install fedoracoreos. Then we would need an --ignition option like with have for --cloud-init, with some sensible defaults. We have a meta issue for that already, please drop more thoughts in there: virt-manager/virt-manager#152

virt-install direct kernel booting doesn't allow the OS to specify default kernel arguments. We can partially work around that with inst.repo, possibly at the cost of redownloading the rootfs. But we'd fail to set the installed platform ID correctly unless the user set it, or e.g. inst.repo engaged an ugly platform-detection hack.

If there's some static default kernel arguments that would go into .treeinfo anyways, IMO save users the step and make the initrd key it off the inst.repo equivalent.

[bgilbert]

We only pass default kernel arguments (inst.repo etc) if we are installing from a network tree. All other cases including local iso we don't pass any arguments.

Okay, that helps.

If we had Fedora CoreOS entries in osinfo-db, which point to stable locations for qcow2 disk images, we could extend virt-install to pull these down automatically virt-install --install fedoracoreos.

SGTM. By design, our only stable URLs at present are for the stream metadata, but the redirector proposed in #625 could address that too.

Is virt-install equipped to handle GPG verification? I'm not seeing GPG keys in the osinfo-db repo.

If there's some static default kernel arguments that would go into .treeinfo anyways, IMO save users the step and make the initrd key it off the inst.repo equivalent.

We currently detect the ISO by searching for a build-specific filesystem label which is embedded in the kernel arguments. If the treeinfo file includes kargs, and we're generating the treeinfo file for every build, we could preserve this property. The goal is to avoid heuristics like "wait 10 seconds for a filesystem that looks vaguely like a CoreOS live image, and then fall back to something else", since they can fail in some cases. I think we could get away with this:

1. Use rootfs from appended initrd if present,
2. else use coreos.live.rootfs_url if present,
3. else use the inst.repo equivalent (coreos.live.tree_url?) if present,
4. else require the ISO.

We'd lose robustness in the edge case that multiple CoreOS ISOs are attached to the system; we'd probably just pick one. We could use the rootfs stream hash to verify that the ISO version matches the initramfs, and fail if not. We'd also need to duplicate the above logic from coreos-livepxe-rootfs.sh into live-generator.

So I think we can work within the current virt-install mechanism if needed; it just seems odd that the direct-boot option takes away the OS's traditional direct control over its kernel arguments. The image-based install would still be preferable as a default, but ideally we'd support both that and the installer flow.

[lucab]

For reference, I cross-linked this into https://gitlab.com/libosinfo/osinfo-db/-/issues/67 in order to see how to approach the "rolling streams with releases" in osinfo-db.

[fidencio]

And, just to make things cleaner, I'm tracking FCOS addition to osinfo-db here: https://gitlab.com/libosinfo/osinfo-db/-/issues/68

[fidencio]

And, just to make things cleaner, I'm tracking FCOS addition to osinfo-db here: https://gitlab.com/libosinfo/osinfo-db/-/issues/68

This PR just got merged and should be part of our next release.

[dustymabe]

Thanks for the great discussion here @bgilbert @crobinso @jlebon @fidencio !

[crobinso]

Is virt-install equipped to handle GPG verification? I'm not seeing GPG keys in the osinfo-db repo.

Neither virt-install or libosinfo have any GPG awareness. It's a good thing to add though.

Everything else sounds good to me to the extent that I understood it :)

[geofreyr]

I think I just discovered this issue when trying to install OKD with coreos 34 and injecting parameters; no rootfs found.