Use a minimal initrd to switch to the full initrd stored in /usr #110
+202
−1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2 tasks
pothos
added a commit
to flatcar/scripts
that referenced
this pull request
Sep 12, 2025
Pulls in flatcar/bootengine#110 Co-authored-by: James Le Cuirot <[email protected]> Signed-off-by: Kai Lueke <[email protected]>
pothos
force-pushed
the
kai/initrd-in-usr
branch
2 times, most recently
from
0316d71 to
6794e4b
Compare
pothos
added a commit
to flatcar/scripts
that referenced
this pull request
Sep 15, 2025
Pulls in flatcar/bootengine#110 Co-authored-by: James Le Cuirot <[email protected]> Signed-off-by: Kai Lueke <[email protected]>
pothos
force-pushed
the
kai/initrd-in-usr
branch
5 times, most recently
from
0dc77a3 to
4dd3627
Compare
pothos
added a commit
to flatcar/scripts
that referenced
this pull request
Sep 15, 2025
Pulls in flatcar/bootengine#110 Co-authored-by: James Le Cuirot <[email protected]> Signed-off-by: Kai Lueke <[email protected]>
pothos
force-pushed
the
kai/initrd-in-usr
branch
2 times, most recently
from
e42505c to
e8672f8
Compare
pothos
added a commit
to flatcar/scripts
that referenced
this pull request
Sep 15, 2025
Pulls in flatcar/bootengine#110 Co-authored-by: James Le Cuirot <[email protected]> Signed-off-by: Kai Lueke <[email protected]>
pothos
added a commit
to flatcar/scripts
that referenced
this pull request
Sep 16, 2025
Pulls in flatcar/bootengine#110 Co-authored-by: James Le Cuirot <[email protected]> Signed-off-by: Kai Lueke <[email protected]>
pothos
force-pushed
the
kai/initrd-in-usr
branch
2 times, most recently
from
3172cf1 to
91da72b
Compare
pothos
added a commit
to flatcar/scripts
that referenced
this pull request
Sep 17, 2025
Pulls in flatcar/bootengine#110 Co-authored-by: James Le Cuirot <[email protected]> Signed-off-by: Kai Lueke <[email protected]>
pothos
added a commit
to flatcar/scripts
that referenced
this pull request
Sep 17, 2025
Pulls in flatcar/bootengine#110 Co-authored-by: James Le Cuirot <[email protected]> Signed-off-by: Kai Lueke <[email protected]>
pothos
force-pushed
the
kai/initrd-in-usr
branch
from
91da72b to
ce52950
Compare
pothos
added a commit
to flatcar/scripts
that referenced
this pull request
Sep 18, 2025
Pulls in flatcar/bootengine#110 Co-authored-by: James Le Cuirot <[email protected]> Signed-off-by: Kai Lueke <[email protected]>
pothos
added a commit
to flatcar/scripts
that referenced
this pull request
Sep 18, 2025
Pulls in flatcar/bootengine#110 Co-authored-by: James Le Cuirot <[email protected]> Signed-off-by: Kai Lueke <[email protected]>
pothos
changed the title
pothos
force-pushed
the
kai/initrd-in-usr
branch
from
ce52950 to
b187cfc
Compare
pothos
added a commit
to flatcar/scripts
that referenced
this pull request
Sep 19, 2025
The growth of binaries over time and the inclusion of new features filled the available boot partition space, so that the kernel+initrd almost couldn't fit twice anymore as required for updates. We employed workarounds such as wrapper scripts for ignition, afterburn and other binaries so that they are loaded from /usr. However, this was still not enough and we would have to do the same for (network) kernel modules and firmware. To avoid making this ever more complex we can use a dedicated initrd focused on loading the full initrd from /usr and then this full initrd can use dracut as before and even drop all the workarounds we accumulated. Generate a minimal initrd to use instead of the full bootengine initrd. The bootengine initrd gets stored as squashfs on /usr. The minimal initrd still includes the early_cpio for amd64 microcode updates. We have a fixed list of modules or module directories to include, only focused on loading /usr and any emergency console interaction. This requires also checking for module dependencies to copy over. The busybox, veritysetup, and kmod binaries are needed and get their required libraries resolved and copied over. They are not static and use shared libraries which should be ok for now. The resulting vmlinuz file is 27 MB for amd64, down from ~60 MB, so we have enough room to include more kernel modules and so on for the next years while we also grow the boot partition and wait for users to redeploy until we can rely on a larger boot partition and eventually drop the minimal initrd again. Pulls in flatcar/bootengine#110 for the minimal initrd script and flatcar/seismograph#12 for making the device mapper discovery for the "rootdev" command more reliable. This also requied a backport of a kernel patch from 2017 that exposes the PARTUUID in the /sys uevent file. Co-authored-by: James Le Cuirot <[email protected]> Signed-off-by: Kai Lueke <[email protected]>
pothos
force-pushed
the
kai/initrd-in-usr
branch
from
b187cfc to
39654dd
Compare
pothos
added a commit
to flatcar/scripts
that referenced
this pull request
Sep 19, 2025
The growth of binaries over time and the inclusion of new features filled the available boot partition space, so that the kernel+initrd almost couldn't fit twice anymore as required for updates. We employed workarounds such as wrapper scripts for ignition, afterburn and other binaries so that they are loaded from /usr. However, this was still not enough and we would have to do the same for (network) kernel modules and firmware. To avoid making this ever more complex we can use a dedicated initrd focused on loading the full initrd from /usr and then this full initrd can use dracut as before and even drop all the workarounds we accumulated. Generate a minimal initrd to use instead of the full bootengine initrd. The bootengine initrd gets stored as squashfs on /usr. The minimal initrd still includes the early_cpio for amd64 microcode updates. We have a fixed list of modules or module directories to include, only focused on loading /usr and any emergency console interaction. This requires also checking for module dependencies to copy over. The busybox, veritysetup, and kmod binaries are needed and get their required libraries resolved and copied over. They are not static and use shared libraries which should be ok for now. The resulting vmlinuz file is 27 MB for amd64, down from ~60 MB, so we have enough room to include more kernel modules and so on for the next years while we also grow the boot partition and wait for users to redeploy until we can rely on a larger boot partition and eventually drop the minimal initrd again. Pulls in flatcar/bootengine#110 for the minimal initrd script and flatcar/seismograph#12 for making the device mapper discovery for the "rootdev" command more reliable. This also requied a backport of a kernel patch from 2017 that exposes the PARTUUID in the /sys uevent file. Co-authored-by: James Le Cuirot <[email protected]> Signed-off-by: Kai Lueke <[email protected]>
The growth of binaries over time and the inclusion of new features filled the available boot partition space, so that the kernel+initrd almost couldn't fit twice anymore as required for updates. We employed workarounds such as wrapper scripts for ignition, afterburn and other binaries so that they are loaded from /usr. However, this was still not enough and we would have to do the same for (network) kernel modules and firmware. To avoid making this ever more complex we can use a dedicated initrd focused on loading the full initrd from /usr and then this full initrd can use dracut as before and even drop all the workarounds we accumulated. Introduce a busybox init script that prepares a minimal environment, has debug toggles and an emergency shell, and only loads the real initrd from /usr to switch over to it. Because mdev is not a proper udev replacement, some additional scripting is needed. Busybox's modprobe can't work with dependencies well and we need the real kmod for that (which is also good to guarantee have the same modprobe options set). Also, some other busybox commands are often lacking things such as loading a kernel module automatically and this has to be done explicitly. We still set up dm-verity for /usr so that we have the same security properties (The code comes from the bootengine systemd generators we have and also covers the PXE boot with a squashfs /usr passed from an additional cpio). The real initrd then reuses the mount point for /usr, and loads any kernel modules and firmware that wasn't loaded already. We also have to make the dependencies for parse-ip-for-networkd.service a bit more explicit because the removal of the /sysusr mount in the full initrd exposed a race condition. Signed-off-by: Kai Lueke <[email protected]>
pothos
force-pushed
the
kai/initrd-in-usr
branch
from
39654dd to
0d8c0af
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The growth of binaries over time and the inclusion of new features
filled the available boot partition space, so that the kernel+initrd
almost couldn't fit twice anymore as required for updates. We employed
workarounds such as wrapper scripts for ignition, afterburn and other
binaries so that they are loaded from /usr. However, this was still not
enough and we would have to do the same for (network) kernel modules and
firmware. To avoid making this ever more complex we can use a dedicated
initrd focused on loading the full initrd from /usr and then this full
initrd can use dracut as before and even drop all the workarounds we
accumulated.
Introduce a busybox init script that prepares a minimal environment,
has debug toggles and an emergency shell, and only loads the real initrd
from /usr to switch over to it. Because mdev is not a proper udev
replacement, some additional scripting is needed. Busybox's modprobe
can't work with dependencies well and we need the real kmod for that
(which is also good to guarantee have the same modprobe options set).
Also, some other busybox commands are often lacking things such as
loading a kernel module automatically and this has to be done
explicitly. We still set up dm-verity for /usr so that we have the same
security properties (The code comes from the bootengine systemd
generators we have and also covers the PXE boot with a squashfs /usr
passed from an additional cpio). The real initrd then reuses the mount
point for /usr, and loads any kernel modules and firmware that wasn't
loaded already.
We also have to make the dependencies for parse-ip-for-networkd.service
a bit more explicit because the removal of the /sysusr mount in the full
initrd exposed a race condition.
How to use
With flatcar/scripts#3241
Testing done
See above