Static GRUB config migration does not handle GRUB configs stored in ESP

[travier]

On older Silverblue systems (installed from Silverblue 33 at least), the GRUB config is stored in the ESP instead of /boot.

The migration logic that we have right now is not enough, as it would have to look at this file as well, clean it up, and enable blscfg and maybe also make sure to add the info pointing GRUB to the boot partition to find the boot entries.

See: https://discussion.fedoraproject.org/t/sb42-updates-fail-on-system-with-512mb-boot-128mb-boot-efi/145041/14

And also: #856

[travier]

It looks like blscfg should be enabled by past updates of this file, so we would only have to cleanup the old ostree entries.

[HuijingHei]

Maybe my understanding is not correct, do we need to run grub2-mkconfig to regenerate the new grub.cfg, also enable blscfg as I see the GRUB_ENABLE_BLSCFG=false in /etc/default/grub? Is it necessary to migrate to the static grub config in this case?

[travier]

Maybe my understanding is not correct, do we need to run grub2-mkconfig to regenerate the new grub.cfg, also enable blscfg as I see the GRUB_ENABLE_BLSCFG=false in /etc/default/grub? Is it necessary to migrate to the static grub config in this case?

We can not use grub2-mkconfig on a system with composefs enabled until https://bugzilla.redhat.com/show_bug.cgi?id=2308594 if fixed. This is why the migration does things manually and does not call it.

When performing the migration for older systems, we should make sure that GRUB_ENABLE_BLSCFG is set to true in /etc/default/grub. This means that we need further manual fixes to the GRUB config as we can not rely on grub2-mkconfig.

[travier]

The initial recommendation that was written for users to remove the duplicated entries was to disable BLS support (https://fedoraproject.org/wiki/Common_F31_bugs#On_Fedora_Silverblue.2FIoT.2C_the_GRUB_menu_shows_duplicate_entries), so this issue might hit a few users.

One option would be to force re-enable the BLS config in a Fedora 40/41 update and hope that users properly update their system to the latest version before rebasing, as we can not re-generate the GRUB config once on F42 with composefs.

[HuijingHei]

To make bootupd robust, we should do to improve:

• enable the BLS config in /etc/default/grub
• find the correct grub.cfg that might existing in ESP (not in /boot)
• remove the duplicated entries in block 15_ostree
• inject blscfg in block 10_linux if BLS config was false

Could you help to review if I missed something? @travier

[travier]

If I understand correctly, the insmod & boot/root options from https://discussion.fedoraproject.org/t/sb42-updates-fail-on-system-with-512mb-boot-128mb-boot-efi/145041/23 would also be missing, so we would have to inject them.

We don't need to touch /etc/default/grub as we won't be re-generating the config anyway.

At this point, this is getting into GRUB config rewrite territory, so I'm not sure if we should do this. But the other alternative is to tell folks to enable the BLS config option back in their config before they update, which does not really scale.

[HuijingHei]

Agree that it will be risky to rewrite config and might make it unable to boot. I have no idea if we have a better / safer way. Any suggestions?

[travier]

An option is to ship a script on F40 & F41 Atomic Desktops & IoT to force this option on at boot. All users that update to the latest F41 before rebasing to F42 would get it, and that should cover the most common case.

[travier]

Fedora IoT systems are using "streams" (stable, testing, rawhide) so the workaround above would not completely cover the case where someone running F41 IoT before that script gets in directly updates to a F42 build.

[travier]

https://pagure.io/workstation-ostree-config/pull-request/630

[HuijingHei]

If there is no such GRUB_ENABLE_BLSCFG=false or even no /etc/default/grub, blscfg is default enabled, right?

[travier]

Adapted from my reply in the PR:

/etc/default/grub should exists on all systems with a dynamic GRUB config. If it does not exists then it should mean that the system has been installed with a static config and should already be have BLS enabled (and thus the migration is mostly a NOP).

According to /etc/grub.d/10_linux, BLS is the default if grubby is not installed:

...
# Make BLS the default if GRUB_ENABLE_BLSCFG was not set and grubby is not installed.
if [ -z "${GRUB_ENABLE_BLSCFG}" ] && ! command -v new-kernel-pkg >/dev/null; then
          GRUB_ENABLE_BLSCFG="true"
fi
...

In the case of https://pagure.io/workstation-ostree-config/pull-request/630, if GRUB_ENABLE_BLSCFG= is not present in /etc/default/grub then the script should add it.

[HuijingHei]

Got it, thanks!