Create tooling to take a FCOS-derived container and generate bootable images from it #1151
Comments
cgwalters
added
kind/enhancement
area/bootable-containers
This is part of coreos/fedora-coreos-tracker#1151 Our generated disk images are largely just a "shell" around the egg of an ostree commit. There is almost nothing that lives in the disk image that isn't in the commit. (This is especially true now that a preparatory commit previous to this moved the *content* of our static `grub.cfg` into `image.json`) In the original coreos-assembler design I'd tried to cleanly separate builds of the ostree from disk image builds, but also support linking them together (with matching version numbers, etc.) The separate `image.yaml` was part of this. This...mostly worked. The goal of this change is that image builds take as input *just the ostree commit*. Crucially we would no longer need the config git repository. And this in turn unlocks truly better separating ostree builds from disk image builds in the pipeline *and* supporting downstream tooling generating disk images from custom containers. One neat thing here is we will finally fix a longstanding issue where coreos-assembler fails when just the `image.yaml` changes: TODO add c-loses for 9-1-2
This is part of coreos/fedora-coreos-tracker#1151 Our generated disk images are largely just a "shell" around the egg of an ostree commit. There is almost nothing that lives in the disk image that isn't in the commit. (This is especially true now that a preparatory commit previous to this moved the *content* of our static `grub.cfg` into `image.json`) In the original coreos-assembler design I'd tried to cleanly separate builds of the ostree from disk image builds, but also support linking them together (with matching version numbers, etc.) The separate `image.yaml` was part of this. This...mostly worked. This change furthers that separation by having image builds input from *just the ostree commit*. Crucially we would no longer need the config git repository to perform an image build. And this in turn unlocks truly better separating ostree builds from disk image builds in the pipeline *and* supporting downstream tooling generating disk images from custom containers. One neat thing here is we will finally fix a longstanding issue where coreos-assembler fails when just the `image.yaml` changes: TODO add c-loses for 9-1-2
This is part of coreos/fedora-coreos-tracker#1151 Our generated disk images are largely just a "shell" around the egg of an ostree commit. There is almost nothing that lives in the disk image that isn't in the commit. (This is especially true now that a preparatory commit previous to this moved the *content* of our static `grub.cfg` into `image.json`) In the original coreos-assembler design I'd tried to cleanly separate builds of the ostree from disk image builds, but also support linking them together (with matching version numbers, etc.) The separate `image.yaml` was part of this. This...mostly worked. This change furthers that separation by having image builds input from *just the ostree commit*. Crucially we would no longer need the config git repository to perform an image build. And this in turn unlocks truly better separating ostree builds from disk image builds in the pipeline *and* supporting downstream tooling generating disk images from custom containers. One neat thing here is we will finally fix a longstanding issue where coreos-assembler fails when just the `image.yaml` changes: TODO add c-loses for 9-1-2
This is part of coreos/fedora-coreos-tracker#1151 Our generated disk images are largely just a "shell" around the egg of an ostree commit. There is almost nothing that lives in the disk image that isn't in the commit. (This is especially true now that a preparatory commit previous to this moved the *content* of our static `grub.cfg` into `image.json`) In the original coreos-assembler design I'd tried to cleanly separate builds of the ostree from disk image builds, but also support linking them together (with matching version numbers, etc.) The separate `image.yaml` was part of this. This...mostly worked. This change furthers that separation by having image builds input from *just the ostree commit*. Crucially we would no longer need the config git repository to perform an image build. And this in turn unlocks truly better separating ostree builds from disk image builds in the pipeline *and* supporting downstream tooling generating disk images from custom containers. One neat thing here is we will finally fix a longstanding issue where coreos-assembler fails when just the `image.yaml` changes: TODO add c-loses for 9-1-2
This is part of coreos/fedora-coreos-tracker#1151 Our generated disk images are largely just a "shell" around the egg of an ostree commit. There is almost nothing that lives in the disk image that isn't in the commit. (This is especially true now that a preparatory commit previous to this moved the *content* of our static `grub.cfg` into `image.json`) In the original coreos-assembler design I'd tried to cleanly separate builds of the ostree from disk image builds, but also support linking them together (with matching version numbers, etc.) The separate `image.yaml` was part of this. This...mostly worked. This change furthers that separation by having image builds input from *just the ostree commit*. Crucially we would no longer need the config git repository to perform an image build. And this in turn unlocks truly better separating ostree builds from disk image builds in the pipeline *and* supporting downstream tooling generating disk images from custom containers. One neat thing here is we will finally fix a longstanding issue where coreos-assembler fails when just the `image.yaml` changes: Closes: coreos#912
This is part of coreos/fedora-coreos-tracker#1151 Our generated disk images are largely just a "shell" around the egg of an ostree commit. There is almost nothing that lives in the disk image that isn't in the commit. (This is especially true now that a preparatory commit previous to this moved the *content* of our static `grub.cfg` into `image.json`) In the original coreos-assembler design I'd tried to cleanly separate builds of the ostree from disk image builds, but also support linking them together (with matching version numbers, etc.) The separate `image.yaml` was part of this. This...mostly worked. This change furthers that separation by having image builds input from *just the ostree commit*. Crucially we would no longer need the config git repository to perform an image build. And this in turn unlocks truly better separating ostree builds from disk image builds in the pipeline *and* supporting downstream tooling generating disk images from custom containers. One neat thing here is we will finally fix a longstanding issue where coreos-assembler fails when just the `image.yaml` changes: Closes: coreos#972
This is part of coreos/fedora-coreos-tracker#1151 Our generated disk images are largely just a "shell" around the egg of an ostree commit. There is almost nothing that lives in the disk image that isn't in the commit. (This is especially true now that a preparatory commit previous to this moved the *content* of our static `grub.cfg` into `image.json`) In the original coreos-assembler design I'd tried to cleanly separate builds of the ostree from disk image builds, but also support linking them together (with matching version numbers, etc.) The separate `image.yaml` was part of this. This...mostly worked. This change furthers that separation by having image builds input from *just the ostree commit*. Crucially we would no longer need the config git repository to perform an image build. And this in turn unlocks truly better separating ostree builds from disk image builds in the pipeline *and* supporting downstream tooling generating disk images from custom containers. One neat thing here is we will finally fix a longstanding issue where coreos-assembler fails when just the `image.yaml` changes: Closes: coreos#972
This is part of coreos/fedora-coreos-tracker#1151 Our generated disk images are largely just a "shell" around the egg of an ostree commit. There is almost nothing that lives in the disk image that isn't in the commit. (This is especially true now that a preparatory commit previous to this moved the *content* of our static `grub.cfg` into `image.json`) In the original coreos-assembler design I'd tried to cleanly separate builds of the ostree from disk image builds, but also support linking them together (with matching version numbers, etc.) The separate `image.yaml` was part of this. This...mostly worked. This change furthers that separation by having image builds input from *just the ostree commit*. Crucially we would no longer need the config git repository to perform an image build. And this in turn unlocks truly better separating ostree builds from disk image builds in the pipeline *and* supporting downstream tooling generating disk images from custom containers. One neat thing here is we will finally fix a longstanding issue where coreos-assembler fails when just the `image.yaml` changes: Closes: coreos#972
This is part of coreos/fedora-coreos-tracker#1151 Our generated disk images are largely just a "shell" around the egg of an ostree commit. There is almost nothing that lives in the disk image that isn't in the commit. (This is especially true now that a preparatory commit previous to this moved the *content* of our static `grub.cfg` into `image.json`) In the original coreos-assembler design I'd tried to cleanly separate builds of the ostree from disk image builds, but also support linking them together (with matching version numbers, etc.) The separate `image.yaml` was part of this. This...mostly worked. This change furthers that separation by having image builds input from *just the ostree commit*. Crucially we would no longer need the config git repository to perform an image build. And this in turn unlocks truly better separating ostree builds from disk image builds in the pipeline *and* supporting downstream tooling generating disk images from custom containers. One neat thing here is we will finally fix a longstanding issue where coreos-assembler fails when just the `image.yaml` changes: Closes: coreos#972
This is part of coreos/fedora-coreos-tracker#1151 Our generated disk images are largely just a "shell" around the egg of an ostree commit. There is almost nothing that lives in the disk image that isn't in the commit. (This is especially true now that a preparatory commit previous to this moved the *content* of our static `grub.cfg` into `image.json`) In the original coreos-assembler design I'd tried to cleanly separate builds of the ostree from disk image builds, but also support linking them together (with matching version numbers, etc.) The separate `image.yaml` was part of this. This...mostly worked. This change furthers that separation by having image builds input from *just the ostree commit*. Crucially we would no longer need the config git repository to perform an image build. And this in turn unlocks truly better separating ostree builds from disk image builds in the pipeline *and* supporting downstream tooling generating disk images from custom containers. One neat thing here is we will finally fix a longstanding issue where coreos-assembler fails when just the `image.yaml` changes: Closes: #972
Retry landing this since a followup commit should fix AWS replication. This is part of coreos/fedora-coreos-tracker#1151 Our generated disk images are largely just a "shell" around the egg of an ostree commit. There is almost nothing that lives in the disk image that isn't in the commit. (This is especially true now that a preparatory commit previous to this moved the *content* of our static `grub.cfg` into `image.json`) In the original coreos-assembler design I'd tried to cleanly separate builds of the ostree from disk image builds, but also support linking them together (with matching version numbers, etc.) The separate `image.yaml` was part of this. This...mostly worked. This change furthers that separation by having image builds input from *just the ostree commit*. Crucially we would no longer need the config git repository to perform an image build. And this in turn unlocks truly better separating ostree builds from disk image builds in the pipeline *and* supporting downstream tooling generating disk images from custom containers. One neat thing here is we will finally fix a longstanding issue where coreos-assembler fails when just the `image.yaml` changes: Closes: coreos#972
This is part of coreos/fedora-coreos-tracker#1151 Our generated disk images are largely just a "shell" around the egg of an ostree commit. There is almost nothing that lives in the disk image that isn't in the commit. (This is especially true now that a preparatory commit previous to this moved the *content* of our static `grub.cfg` into `image.json`) In the original coreos-assembler design I'd tried to cleanly separate builds of the ostree from disk image builds, but also support linking them together (with matching version numbers, etc.) The separate `image.yaml` was part of this. This...mostly worked. This change furthers that separation by having image builds input from *just the ostree commit*. Crucially we would no longer need the config git repository to perform an image build. And this in turn unlocks truly better separating ostree builds from disk image builds in the pipeline *and* supporting downstream tooling generating disk images from custom containers. One neat thing here is we will finally fix a longstanding issue where coreos-assembler fails when just the `image.yaml` changes: Closes: coreos#972
This is part of coreos/fedora-coreos-tracker#1151 Our generated disk images are largely just a "shell" around the egg of an ostree commit. There is almost nothing that lives in the disk image that isn't in the commit. (This is especially true now that a preparatory commit previous to this moved the *content* of our static `grub.cfg` into `image.json`) In the original coreos-assembler design I'd tried to cleanly separate builds of the ostree from disk image builds, but also support linking them together (with matching version numbers, etc.) The separate `image.yaml` was part of this. This...mostly worked. This change furthers that separation by having image builds input from *just the ostree commit*. Crucially we would no longer need the config git repository to perform an image build. And this in turn unlocks truly better separating ostree builds from disk image builds in the pipeline *and* supporting downstream tooling generating disk images from custom containers. One neat thing here is we will finally fix a longstanding issue where coreos-assembler fails when just the `image.yaml` changes: Closes: coreos#972
This is part of coreos/fedora-coreos-tracker#1151 Our generated disk images are largely just a "shell" around the egg of an ostree commit. There is almost nothing that lives in the disk image that isn't in the commit. (This is especially true now that a preparatory commit previous to this moved the *content* of our static `grub.cfg` into `image.json`) In the original coreos-assembler design I'd tried to cleanly separate builds of the ostree from disk image builds, but also support linking them together (with matching version numbers, etc.) The separate `image.yaml` was part of this. This...mostly worked. This change furthers that separation by having image builds input from *just the ostree commit*. Crucially we would no longer need the config git repository to perform an image build. And this in turn unlocks truly better separating ostree builds from disk image builds in the pipeline *and* supporting downstream tooling generating disk images from custom containers. One neat thing here is we will finally fix a longstanding issue where coreos-assembler fails when just the `image.yaml` changes: Closes: coreos#972
This is part of coreos/fedora-coreos-tracker#1151 Our generated disk images are largely just a "shell" around the egg of an ostree commit. There is almost nothing that lives in the disk image that isn't in the commit. (This is especially true now that a preparatory commit previous to this moved the *content* of our static `grub.cfg` into `image.json`) In the original coreos-assembler design I'd tried to cleanly separate builds of the ostree from disk image builds, but also support linking them together (with matching version numbers, etc.) The separate `image.yaml` was part of this. This...mostly worked. This change furthers that separation by having image builds input from *just the ostree commit*. Crucially we would no longer need the config git repository to perform an image build. And this in turn unlocks truly better separating ostree builds from disk image builds in the pipeline *and* supporting downstream tooling generating disk images from custom containers. One neat thing here is we will finally fix a longstanding issue where coreos-assembler fails when just the `image.yaml` changes: Closes: coreos#972
This is part of coreos/fedora-coreos-tracker#1151 Our generated disk images are largely just a "shell" around the egg of an ostree commit. There is almost nothing that lives in the disk image that isn't in the commit. (This is especially true now that a preparatory commit previous to this moved the *content* of our static `grub.cfg` into `image.json`) In the original coreos-assembler design I'd tried to cleanly separate builds of the ostree from disk image builds, but also support linking them together (with matching version numbers, etc.) The separate `image.yaml` was part of this. This...mostly worked. This change furthers that separation by having image builds input from *just the ostree commit*. Crucially we would no longer need the config git repository to perform an image build. And this in turn unlocks truly better separating ostree builds from disk image builds in the pipeline *and* supporting downstream tooling generating disk images from custom containers. One neat thing here is we will finally fix a longstanding issue where coreos-assembler fails when just the `image.yaml` changes: Closes: coreos#972
This is part of coreos/fedora-coreos-tracker#1151 Our generated disk images are largely just a "shell" around the egg of an ostree commit. There is almost nothing that lives in the disk image that isn't in the commit. (This is especially true now that a preparatory commit previous to this moved the *content* of our static `grub.cfg` into `image.json`) In the original coreos-assembler design I'd tried to cleanly separate builds of the ostree from disk image builds, but also support linking them together (with matching version numbers, etc.) The separate `image.yaml` was part of this. This...mostly worked. This change furthers that separation by having image builds input from *just the ostree commit*. Crucially we would no longer need the config git repository to perform an image build. And this in turn unlocks truly better separating ostree builds from disk image builds in the pipeline *and* supporting downstream tooling generating disk images from custom containers. One neat thing here is we will finally fix a longstanding issue where coreos-assembler fails when just the `image.yaml` changes: Closes: #972
This is part of coreos/fedora-coreos-tracker#1151 Our generated disk images are largely just a "shell" around the egg of an ostree commit. There is almost nothing that lives in the disk image that isn't in the commit. (This is especially true now that a preparatory commit previous to this moved the *content* of our static `grub.cfg` into `image.json`) In the original coreos-assembler design I'd tried to cleanly separate builds of the ostree from disk image builds, but also support linking them together (with matching version numbers, etc.) The separate `image.yaml` was part of this. This...mostly worked. This change furthers that separation by having image builds input from *just the ostree commit*. Crucially we would no longer need the config git repository to perform an image build. And this in turn unlocks truly better separating ostree builds from disk image builds in the pipeline *and* supporting downstream tooling generating disk images from custom containers. One neat thing here is we will finally fix a longstanding issue where coreos-assembler fails when just the `image.yaml` changes: Closes: coreos#972
This is part of coreos/fedora-coreos-tracker#1151 Our generated disk images are largely just a "shell" around the egg of an ostree commit. There is almost nothing that lives in the disk image that isn't in the commit. (This is especially true now that a preparatory commit previous to this moved the *content* of our static `grub.cfg` into `image.json`) In the original coreos-assembler design I'd tried to cleanly separate builds of the ostree from disk image builds, but also support linking them together (with matching version numbers, etc.) The separate `image.yaml` was part of this. This...mostly worked. This change furthers that separation by having image builds input from *just the ostree commit*. Crucially we would no longer need the config git repository to perform an image build. And this in turn unlocks truly better separating ostree builds from disk image builds in the pipeline *and* supporting downstream tooling generating disk images from custom containers. One neat thing here is we will finally fix a longstanding issue where coreos-assembler fails when just the `image.yaml` changes: Closes: #972
|
I briefly looked at trying to have a new tool that "restamps" an existing live ISO with new content by scraping out the files, but what makes this hard right now is that https://github.com/coreos/coreos-assembler/blob/main/src/cmd-buildextend-live has a whole lot of load-bearing magic around things like the kernel arguments embed area and architecture specifics. I am pretty sure it'd be possible to write a tool which basically just dropped in a new squashfs into an existing ISO and kept everything else about it. That'd allow overriding the rootfs...but would break when trying to override the kernel, which is what most use cases for this want anyways, so probably not worth trying. Further, that code specifically also reads |
|
Been thinking about this more. How about this plan:
|
|
@cgwalters, Regarding #1151 (comment): If we are re-generating the ISO using the coreos-installer then why is it better than using the coreos-assembler for it? I though that the benefit of the installer is not manipulate the ISO without rebuilding it from source each time. |
|
My latest work on this is in containers/bootc#30 |
|
@cgwalters is the thought that having an |
|
The initial design of Further, one wants to use the same technology stack when fetching updates (aside from writing the partitions etc.). This is why I created the new bootc project, which is exactly this. I would like to add bootc to FCOS, and am working in that direction. |
|
@cgwalters that makes a lot of sense. Do you see bootc also sidestepping the need to add functionality to ignition/butane to configure an installed system to boot a containerized build? Such as what is hinted at in the coreos layering enhancements document. I'm assuming that what is described at the end of that document is not currently available functionality. I did not see it mentioned in the latest butane documentation. |
Yes...and no. Yes, |
|
OK copy-pasting a comment from rpm-ostree here: We already ship what we need here. I just tried this out: The error here is very revealing - it's because we want to grow the rootfs on firstboot. Which goes right into the design rationale for bootc install making the filesystems directly. Adding in a (Then rebooting I'm in a So.......the question is, short term, do we try to also graft support for this into coreos-installer directly to improve the ergonomics? For Fedora+derivatives, running this code pulls in In practice I think I'd vote for a soft dep because I generally only care about making this work from inside an existing FCOS-derivative live environment, where we already have the requisite depchain. The alternative here is to just document things. I'm fine with that too. |
|
Also though of note, this flow is only helpful for the scenario where one can get a machine up enough to run this code and write to disk. There are some cases where we can e.g. attach a stock ISO via IPMI or so, then do this installation to disk with the real NIC drivers already embedded etc. Or, plugging in a USB stick (say the FCOS-derivative Live ISO) but also with a data partition containing the target container image. It'd also be possible, albeit hacky to e.g. inject ignition into our live ISO which downloads and loads the kmod dynamically separately. (Or I guess...assuming the live ISO kernel is the same as the container kernel, could actually extract it from the container) Hmmm....I wonder though, maybe short term we could try to make some tool which actually does the "data partition with container" thing more streamlined. That would help with a few more cases perhaps? But ultimately we do need some sort of ergonomic tool which takes a container image as input and spits out an ISO that boots using the modified kernel/OS and can install itself to disk using that target container image. |
|
BTW though, there are still a lot of architectural problems with the "hackily update after coreos-installer" approach. For example, because things like LUKS are handled via Ignition always today, it means that the files we write from the container image are unencrypted. For Fedora CoreOS derivatives, this doesn't matter because everything we ship is FOSS - there is no secret content. I think that's not going to be true in the general case for custom derived images; I'm pretty sure some users will embed secret data into images (passwords or keys). We can't really say it's a bad idea to do that unless we offer them a significantly more ergnonomic/compelling way to do it. (This does touch on bootc configmap/secret support some, particularly if we do something nice with secrets). I think Ignition has always had this problem, but Ignition configs also happen on firstboot today and the files it writes will happen after LUKS has been enabled. |
|
I would say as a downstream user/consumer of whatever functionality is developed, a way to directly install a container image to disk explicitly would be my preferred workflow. I have thought about doing the "post install rebase/deploy hack" but it really does seem like exactly that and I'm apt to avoid such hacks at any scale. I think having a simple utility without extensive dependencies that could generate the raw bootable disk image (could this step require qemu virtualization? that would be a drag) that could be consumed by As for ignition, I think if there's a utility to create disk images and ISOs that can be used by the installer to be written to disk (or booted over the network?!) then there is likely little need for support in ignition. If ignition has to do the rebase or deploy that implies a plain FCOS image is installed, booted, ignition runs and then installs a customized container image. This seems terribly inefficient and convoluted. Ultimately I think bootc is my true answer here so hopefully that project matures, stabilizes and sees good adoption. |
|
Just to xref, there is a new "more incremental" potential path here https://issues.redhat.com/browse/OCPBU-66?focusedId=22251336&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-22251336 that's being discussed in the OCP context but would apply to FCOS too |
|
We discovered an ostree bug here: ostreedev/ostree-rs-ext#503 |
|
@cgwalters I tried to implement what you advised in this comment, but the flow failed. Here is what I did:
The ostree command was invoked in the following manner: After reboot, it seems there was an attempt to apply ignition, but there were errors, and the console dropped to emergency shell. Here are some errors that appeared in the console: What is the reason for the failure? How can it be fixed ? |
Hmm that looks like #1250 |
|
Use |
|
For doing OS updates at coreos-installer time, the topic of kernel arguments came up (both Ignition and MachineConfig for OCP). Basically we need to extract the kernel arguments from the Ignition/MC and apply them at this time too. |
|
@cgwalters I implemented #1151 (comment) and it works, except that when I moved to 4.14 image I got error from ostree: When I disabled selinux (setenfoce 0), the ostree command was successful. |
Yes...it's possible but tricky to handle this. We do it in bootc today, see https://github.com/containers/bootc/blob/0a657ea9609e33c09e8bf2c013699c379f0ccc43/lib/src/lsm.rs#L32 But |
|
I am hoping to build a pxe image with zfs added. My intention is to run the os from ram only and never install it onto disk. I think it would be helpful to have an example that uses https://github.com/coreos/layering-examples/tree/main/build-zfs-module to demonstrate creating a bootable image that has zfs installed. Has this issued progressed far enough to do this now? If so could you please show me how I would build one? Thanks for all your effort solving this problem. |
This is touched on in https://github.com/coreos/enhancements/blob/main/os/coreos-layering.md#use-of-coreos-diskboot-images
Basically I think our default stance is that admins use our golden images + ignition to target their custom containers, but I think for a variety of reasons we will want to support generating disk images directly too.
I think this would be relatively maintainable for bare metal targets in particular (ISO, PXE etc.), and it's on bare metal targets where there's more of a desire for things like handling the first boot with an out of tree NIC driver, etc.
Now, obviously this is basically what coreos-assembler is today. However, a few key points:
cosa init --ostree docker://quay.io/coreos-assembler/fcos:testing-develcoreos-assembler#2685This build tool could derive from the coreos-assembler container literally, or we could split out the bits we need as a shared library or so (could make sense as a Go library actually, even if we happen to ship shell scripts and python in it?). Starting out this way is going to be the most supportable thing. This tool should be runnable as a container image (that yes, accepts a container image as input) by default, but we could support other flows too.
That said, we should also do a spike on having e.g. RHEL Image Builder be able to output an ISO at least.
The text was updated successfully, but these errors were encountered: