Use bootupd and remove ostree-grub2 (was: All deployments are shown twice in grub)

[YaLTeR]

F33, ThinkPad T495s

[felipeborges]

I can reproduce this too.

[jlebon]

See https://fedoraproject.org/wiki/Common_F31_bugs#On_Fedora_Silverblue.2FIoT.2C_the_GRUB_menu_shows_duplicate_entries (I've added an "Update:" section there).

[YaLTeR]

See https://fedoraproject.org/wiki/Common_F31_bugs#On_Fedora_Silverblue.2FIoT.2C_the_GRUB_menu_shows_duplicate_entries (I've added an "Update:" section there).

So... should I migrate to BLS? If I should then maybe it should rather be done by default in Silverblue?

[jlebon]

So... should I migrate to BLS? If I should then maybe it should rather be done by default in Silverblue?

If you can, yes. I'm not sure if it's worth forcing a migration at this point because of the potential risks (see e.g. https://discussion.fedoraproject.org/t/boot-entries-gone-after-upgrade/8026). And as mentioned in the wiki, the migration script only works on EFI because it might not be safe to do it automatically on BIOS (see ostreedev/ostree#2044 (comment)).

[Jmennius]

What about systems that did not upgrade to F31 (clean install F31+)?
I did not find any place that creates /boot/grub2/.grub2-blscfg-supported for new installations.
I have BLS enabled (Silverblue) but there is no such file (probably this system started with F31).

I am also concerned that .grub2-switch-to-blscfg does not work well with https://fedoraproject.org/wiki/Changes/UnifyGrubConfig.

cc @jlebon @martinezjavier

[dustymabe]

I'm seeing this on a fresh install of F35 Silverblue. I guess we need to track down what used to create the /boot/grub2/.grub2-blscfg-supported and see why it's not getting created any longer.

[dustymabe]

Found it: https://src.fedoraproject.org/rpms/grub2/blob/9fdaa794e06846f1426bd15360401a8b1a5017a9/f/0065-Add-grub2-switch-to-blscfg.patch#_362

[jlebon]

I'm seeing this on a fresh install of F35 Silverblue. I guess we need to track down what used to create the /boot/grub2/.grub2-blscfg-supported and see why it's not getting created any longer.

I think... nothing ever actually created that for new installs. The idea we discussed was to have Anaconda do it, but it looks like we never actually followed through with it.

The other option is to just assume GRUB is new enough, but that can break people upgrading from an old install with old GRUB. The failure mode here is particularly bad: they'll get no boot entries.

Maybe we just need to be really loud about it and say that you need to make sure that your GRUB is up to date before $date using either the migration script or bootupd, and then we rip out this check.

[jlebon]

Maybe we just need to be really loud about it and say that you need to make sure that your GRUB is up to date before $date using either the migration script or bootupd, and then we rip out this check.

A prerequisite for that is of course to ship bootupd in FSB, which we currently don't.

[jlebon]

Yeah, I think bootupd is probably the cleanest path forward for this and for possible future issues like it. So my strawman is something like:

• add bootupd in FSB
• teach Anaconda to call out to bootupctl
• teach 15_ostree to read bootupd metadata to know if GRUB supports blscfg (but really, if bootupd metadata exists, it surely does, so this might just be test -f /boot/.bootupd-state.json)
• tell upgrading users that this bug is fixed in the latest bootloader, and they can update using bootupctl adopt-and-update

@cgwalters @martinezjavier WDYT?

[travier]

This might need a Fedora Change for F36 (Silverblue & Kinoite).

[martinezjavier]

@cgwalters @martinezjavier WDYT?

This makes sense to me. In fact, it would be great if even the non-ostree Fedora variants could adopt bootupd.

[travier]

The change deadline is mid January if we think this is a self contained change: https://fedorapeople.org/groups/schedule/f-36/f-36-key-tasks.html

[ormandj]

This would be quite a nice change - I was surprised to see the duplicate entries on a clean 35 Silverblue installation, and ran across this issue while researching resolution.

[travier]

How do all of you feel about trying to make that happen for F37 and maybe opt-in for F36 (ship bootupd but not enabled by default)? I can start the change request.

[tpopela]

It was great to see @martinezjavier 's option and if he's in favor, then I'm as well. Thank you @travier for looking into this!

[jlebon]

How do all of you feel about trying to make that happen for F37 and maybe opt-in for F36 (ship bootupd but not enabled by default)? I can start the change request.

SGTM, thanks for picking this up!

[tpopela]

It would be good to tackle this after the recent round of the grub2 CVEs. We might also need to fix coreos/bootupd#108 . But it looks like there was some work done by @cgwalters , but might be not finished - what's the current state of it Colin?

[koko-ng]

I ran into this problem while debugging why firmware updates weren't applied. If anyone runs into the same issue, before there is a cleaner fix it's possible to update shim like this:

sudo rpm-ostree usroverlay 
wget https://kojipkgs.fedoraproject.org//packages/shim/15.6/1/x86_64/shim-x64-15.6-1.x86_64.rpm
sudo rpm -i  --reinstall shim-x64-15.6-1.x86_64.rpm

[dustymabe]

I'm not sure if rpm-ostree usroverlay is what you want:

       usroverlay
           Mount a writable overlay filesystem on /usr which is
           active only for the remainder of the system boot. This
           is intended for development, testing, and debugging.
           Changes will not persist across upgrades, or rebooting
           in general.

[koko-ng]

It's just so that rpm can write to the database, of course it will be discarded on next reboot, but the changes to /bin/efi will effectively be kept as they are not managed by ostree and as rpm reinstall the same version of the package, the content of the persistent database (the one managed by rpm-ostree) is now coherent with the content of /bin/efi.

[travier]

With https://pagure.io/workstation-ostree-config/pull-request/288, we now have bootupd in Rawhide but not enabled by default.

Let's track what's missing here and make a Change request for F38.

For reference, the commit message from Jonathan:

We need to make it easier to update the bootloader on these variants
because unlike on traditional systems, it's not updated automatically
with the rest of the system. Add bootupd for that.

This would allow fixing issues like:
- https://github.com/coreos/rpm-ostree/issues/3715
- https://github.com/fedora-silverblue/issue-tracker/issues/120#issuecomment-979268679

It won't be enabled by default and as mentioned in that comment requires
work in Anaconda to be seamless. But at least with this users should be
able to adopt and update:

https://github.com/coreos/bootupd/blob/main/README-design.md

See also the tracker issue where we did this for Fedora CoreOS:

https://github.com/coreos/fedora-coreos-tracker/issues/510

[Verhoeckx]

@yann-soubeyrand

I executed the three lines that you mention in the above comment and when I now execute (after a reboot) rpm-ostree upgrade, I get the following error: error: loading sysroot: opendirat: Permission denied. Do I need the execute some other command after the three lines you mention (to close the overlay)?

[juhp]

Is this still in current Silverblue/Kinoite? I am not seeing it in either 39 or Rawhide, but if so what to look for?

[travier]

This is still planned to be included in F39 but not there yet due to https://pagure.io/workstation-ostree-config/pull-request/330#comment-190406 (the PR has been reverted).

The test images (gitlab.com/fedora/ostree/ci-test) include this already. The Universal Blue images should as well.

[travier]

Pushed back to Fedora 40. Might come as an update to F39 if we backport the support.

[travier]

bootupd support in Anaconda got merged 🎉

• payload/rpmostree: Add support for bootupd rhinstaller/anaconda#5298 -> Automatic bootupd support rhinstaller/anaconda#5342

[0rzech]

I executed the three lines that you mention in the above comment and when I now execute (after a reboot) rpm-ostree upgrade, I get the following error: error: loading sysroot: opendirat: Permission denied. Do I need the execute some other command after the three lines you mention (to close the overlay)?

@Verhoeckx Did you find any solution to this? I've just stumbled upon this error too.

@travier Is this a problem? While I got the same error, rpm-ostree status did display new deployment as usual. If it is a problem, how to fix these permissions?

[travier]

See #543 (comment) for a lightly tested set of commands to update your EFI bootloader until we have full bootupd support in Fedora Atomic Desktops.

[scottwillmoore]

I'm confused 😕, did bootupd make it into 40, or was it deferred until 41? At least from the threads below it looks like it was pushed back to 41, but then was ready for 40, however I don't see bootupd on my installation?

• https://fedoraproject.org/wiki/Changes/FedoraSilverblueBootupd
• https://bugzilla.redhat.com/show_bug.cgi?id=2150982

Under the assumption bootupd is not included with 40, is it possible to still use it?

[davidstrauss]

I researched that exact question about deferral earlier today. I found at least one comment (I think on Bugzilla) that confirmed deferral to F41. I also noticed that most F40 docs fail to reflect that.

[travier]

I've updated everything to point to F41 as we did not land it in F40. We'll try to backport things to F40 as much as possible.

[Malix-Labs]

Hello @travier, any update about the backport to F40?

[travier]

We can only consider backporting to F40 after it lands in F41. Status is tracked in https://gitlab.com/fedora/ostree/sig/-/issues/1

[travier]

With bootupd landing in Fedora 41, the first part of this issue is done. Now we have to wait for F42 to be able to remove the ostree-grub2 package to complete this.

[davidstrauss]

Nice work, folks. I just updated to Fedora Silverblue 41, and it looks like bootupd is properly active:

straussd@phoenix:~$ sudo bootupctl adopt-and-update
Running as unit: bootupd.service
No components are adoptable.
straussd@phoenix:~$ sudo bootupctl status
Running as unit: bootupd.service
Component EFI
  Installed: grub2-efi-ia32-1:2.12-10.fc41.x86_64,grub2-efi-x64-1:2.12-10.fc41.x86_64,shim-ia32-15.8-3.x86_64,shim-x64-15.8-3.x86_64
  Update: At latest version
No components are adoptable.
Boot method: EFI