nixos/systemd-boot: Add mirroredBoots #246897

Gerg-L · 2023-08-03T03:19:28Z

Description of changes

adds boot.loader.systemd-boot.mirroredBoots

only renamed efiSysMountPoint to mountPoint in the python script

Things done

0x4A6F · 2023-08-03T21:53:06Z

Only tested with systemd-boot and booted from both disks:

   # Use the systemd-boot EFI boot loader.
   boot.loader.systemd-boot.enable = true;
   boot.loader.efi.canTouchEfiVariables = true;

  # https://github.com/NixOS/nixpkgs/pull/246897
  boot.loader.systemd-boot.mirroredBoots = [
    "/boot0"
    "/boot4"
  ];

Any plans to incorporate this into nixosTests?

And what is the reason to rename efiSysMountPoint?

RaitoBezarius · 2023-08-03T22:02:41Z

I can really accept this only with a nixosTest, the code looks pretty much good though.

Gerg-L · 2023-08-03T22:54:55Z

Only tested with systemd-boot and booted from both disks:
   # Use the systemd-boot EFI boot loader.
   boot.loader.systemd-boot.enable = true;
   boot.loader.efi.canTouchEfiVariables = true;

  # https://github.com/NixOS/nixpkgs/pull/246897
  boot.loader.systemd-boot.mirroredBoots = [
    "/boot0"
    "/boot4"
  ];
Any plans to incorporate this into nixosTests?

And what is the reason to rename efiSysMountPoint?

Renamed efiSysMountPoint because the name was no longer accurate to boot.loader.efi.efiSysMountPoint

Gerg-L · 2023-08-03T23:30:39Z

I can really accept this only with a nixosTest, the code looks pretty much good though.

i'll try to get to it tonight

Gerg-L · 2023-08-05T02:02:03Z

I can really accept this only with a nixosTest, the code looks pretty much good though.

yeah I haven't been able to figure out how to make a nixosTest with a custom partition scheme setup before the bootloader is installed

Gerg-L · 2023-08-06T01:11:28Z

@RaitoBezarius can you take a look at the second commit I just pushed, I feel like I'm so close to it working but I can't figure out exactly what I'm doing wrong

nikstur · 2023-08-06T11:55:40Z

This reminds me of #226692 with a slightly different scope.

However I'm not sure I love the idea of mirrored boot. It seems like there is a potential for side effects by calling bootctl install multiple times as it would manipulate efivars each time.

Also see this comment about the dangers of doing so: #152155 (comment)

Gerg-L · 2023-08-06T12:09:55Z

Also see this comment about the dangers of doing so: #152155 (comment)

That's talking about doing mdadm raid1 on your ESP...
This is simply installing the boot loader to multiple partitions

However I'm not sure I love the idea of mirrored boot. It seems like there is a potential for side effects by calling bootctl install multiple times as it would manipulate efivars each time.

I mean there's also boot.loader.grub.mirroredBoots

nikstur · 2023-08-06T13:12:23Z

The linked thread is still insightful:

But just doing mirroring from linux side is not sufficient. The firmware/boot loader can write to the ESP too, and since they don't do RAID things will fall apart badly. And yes, sd-boot writes to the ESP if boot counting/boot assessment is enabled.

On top of that, the ESP is shared between OSes, so as soon as you do multi boot things fall apart even worse.

systemd/systemd#12468 (comment)

mirroredBoots would suffer from the same issue even if its not RAID.

This also means that just by virtue of implementing this bespoke feature we will make our lives harder keeping up with systemd updates since they have already signalled that mirroring the ESP is not something they even consider.

Couldn't you implement this more cleanly by writing a systemd service that checks for file changes on the ESP and then copies all files to the configured locations? This would also take care of multi boot, boot counting etc.

I mean there's also boot.loader.grub.mirroredBoots

I believe grub.mirroredBoot came from the MBR era where this somewhat made sense but they just adapted it to UEFI. Anyways, we shouldn't let what we have done with Grub lead our design. This will not lead to a good place.

Gerg-L · 2023-08-06T13:52:43Z

I completely agree with not following grub

A systemd service should work, but that kinda goes against the use case of mirroring the ESP: Booting with complete redundancy
Then you have a "master" esp and "slave" esp(s)

bjornfor · 2023-08-06T18:13:57Z

systemd/systemd#12468 (comment)

mirroredBoots would suffer from the same issue even if its not RAID.

Why? AFAIU, mirroredBoots suffer from none of these issues.

But I would double check how bootctl install works wrt. the EFI variables, like mentioned before in this thread.

Couldn't you implement this more cleanly by writing a systemd service that checks for file changes on the ESP and then copies all files to the configured locations? This would also take care of multi boot, boot counting etc.

Why would that be any better? That seems just like a more brittle way to implement mirroredBoots. Why do you think boot counting breaks with mirroredBoots?

nikstur · 2023-08-06T20:14:35Z

Why do you think boot counting breaks with mirroredBoots?

The issue is that if the bootloader or another OS (when you dual boot) modifies files on the ESP, these modifications are not accounted for when you mirror the ESP via the mechanism from this PR. If you just rsync the entire ESP to a different location, you also copy these modifications.

I'd also wager that rsyncing your ESP is actually less brittle because it just copies things instead of calling a stateful installation procedure repeatedly.

Booting with complete redundancy

As fas as I can tell, your implementation doesn't guarantee that all ESPs are written in a single transaction (i.e. when writing to one fails, others might still be written to). So then that's not better than having a service that (within a few seconds of writing to the main ESP) copies the main ESP to the backups. I'd argue both ways have the same (weak) guarantees on redundancy.

Gerg-L · 2023-08-06T23:24:47Z

The issue is that if the bootloader or another OS (when you dual boot) modifies files on the ESP, these modifications are not accounted for when you mirror the ESP via the mechanism from this PR. If you just rsync the entire ESP to a different location, you also copy these modifications.

not if the modifications are done to the slave...

bjornfor · 2023-08-09T07:04:46Z

The issue is that if the bootloader or another OS (when you dual boot) modifies files on the ESP, these modifications are not accounted for when you mirror the ESP via the mechanism from this PR. If you just rsync the entire ESP to a different location, you also copy these modifications.

I'm thinking the exact opposite.

I think syncing the boot counting or other state across multiple ESPs somewhat defeats the purpose -- if all boot attempts are are spent on device A, you want firmware to try device B next, but if both devices have the same state now (thanks to background rsync) I think firmware will not attempt device B because it too seems to have spent all boot attempts.

With a systemd service / rsync I'd also expect difficulties with bidirectional syncing (already mentioned in this thread) and that the time window where the mirrored devices are out of sync increases.

My mental model of the ESP is it only changes when NixOS boot configuration changes, and firmware can have a bit of state on the side (not affecting NixOS). That means (1) we only need to write to the ESP when NixOS changes (no background sync needed), and (2) we shouldn't mess with the firmware state (backgrouns sync harmful). I think this is what this PR does.

xaverdh · 2023-08-09T07:56:59Z

The issue is that if the bootloader or another OS (when you dual boot) modifies files on the ESP, these modifications are not accounted for when you mirror the ESP via the mechanism from this PR. If you just rsync the entire ESP to a different location, you also copy these modifications.

Yes and this is precisely what you do not want. With this pr, the information gets duplicated from a source that is known to be intact. When you rsync the existing boot partition, any error present there (e.g. due to corruption of the disk) will propagate to the second copy.

Gerg-L · 2023-08-09T09:51:53Z

We could always go with the nuclear option...

Wipe each ESP every time

Edit: this is a joke... Mostly

0x4A6F · 2023-08-09T18:25:46Z

Getting following with a deploy-rs run on activation:

⭐ ℹ️ [activate] [INFO] Activating profile
activating the configuration...
setting up /etc...
reloading user units for root...
setting up tmpfiles
sed: can't read /boot/loader/loader.conf: No such file or directory
⭐ ❌ [activate] [ERROR] The activation script resulted in a bad exit code: Some(2)
Connection to 192.168.1.42 closed.
🚀 ❌ [deploy] [ERROR] Activating over SSH resulted in a bad exit code: Some(1)
🚀 ℹ️ [deploy] [INFO] Revoking previous deploys
🚀 ❌ [deploy] [ERROR] Deployment failed, rolled back to previous generation

ElvishJerricco · 2023-08-10T20:47:28Z

We could always go with the nuclear option...

Wipe each ESP every time

Edit: this is a joke... Mostly

You joke but... #226168

ElvishJerricco · 2023-08-10T20:50:54Z

Another thing to note is the "random seed": https://systemd.io/RANDOM_SEEDS/

The systemd-boot EFI boot loader included in systemd is able to maintain and provide a random seed stored in the EFI System Partition (ESP) to the booted OS, which allows booting up with a fully initialized entropy pool from earliest boot on. During installation of the boot loader (or when invoking bootctl random-seed) a seed file with an initial seed is placed in a file /loader/random-seed in the ESP. In addition, an identically sized randomized EFI variable called the ‘system token’ is set, which is written to the machine’s firmware NVRAM. During boot, when systemd-boot finds both the random seed file and the system token they are combined and hashed with SHA256 (in counter mode, to generate sufficient data), to generate a new random seed file to store in the ESP as well as a random seed to pass to the OS kernel. The new random seed file for the ESP is then written to the ESP, ensuring this is completed before the OS is invoked.

(Emphasis mine)

This is actually security critical. sd-boot will only update the random seed on the ESP that it's booting. If we then boot a different ESP that hasn't had its random seed updated, then we're reusing an old seed on a new boot, which is very bad practice.

Gerg-L · 2023-08-10T21:15:09Z

Isn't that only if you rsync though?
If you're installing normally to two ESP's that shouldn't be a problem?

nixos-discourse · 2023-11-20T23:12:06Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/zfs-systemd-boot/29956/11

bjornfor · 2023-11-21T08:02:18Z

nixos/modules/system/boot/loader/systemd-boot/systemd-boot.nix

+    installDirs =
+      if cfg.mirroredBoots != []
+      then cfg.mirroredBoots
+      else [efi.efiSysMountPoint];


My first thought was that mirroredBoots should be additional mountpoints, but seeing this I guess the implementation is only these mountpoints (the "main" ESP mountpoint gets ignored)? Am I the only one that find that surprising?

Indeed, and AFAIU the grub.nix case, mirroredBoots are additionals mountpoints there:

nixpkgs/nixos/modules/system/boot/loader/grub/grub.nix

Lines 708 to 710 in ec6cc9c

boot.loader.grub.mirroredBoots = optionals (cfg.devices != [ ]) [

{ path = "/boot"; inherit (cfg) devices; inherit (efi) efiSysMountPoint; }

];

ju1m · 2023-11-22T00:55:42Z

nixos/modules/system/boot/loader/systemd-boot/systemd-boot.nix

+      else [efi.efiSysMountPoint];
+  in
+    pkgs.writeShellScript "install-systemd-boot.sh"
+      (lib.concatMapStrings (x: "${checkedSystemdBootBuilder x} \"$@\"\n") installDirs)


I think that, like in grub.nix, the usual error detection set -e should be activated, and maybe also set -u.

Why keeping on passing $@? I don't see any argument neither used, nor passed (so far).

Nitpick: here each mountpoint generates a new derivation, the mountpoint could instead be passed as an envvar mountpoint=${escapeShellArg x} ${checkedSystemdBootBuilder}. Again that's likely just a few derivations in practice, so just a nitpick.

ju1m · 2023-11-22T01:01:12Z

nixos/modules/system/boot/loader/systemd-boot/systemd-boot.nix

@@ -238,6 +245,17 @@ in {
      '';
    };

+    mirroredBoots = lib.mkOption {
+      type = lib.types.listOf lib.types.str;


grub.nix's mirroredBoots' type is more sophisticated, maybe it would be more correct to use the same or a subset.

Gerg-L · 2023-12-10T22:12:15Z

nikstur is very clearly against this, so I'm just going to close this and keep using grub for now

nixos-discourse · 2024-09-11T07:30:31Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/disko-partition-setup-on-uefi-system-which-has-fallback-boot-partition/51968/1

nixos/systemd-boot: Add mirroredBoots

4418de4

Gerg-L requested a review from dasJ as a code owner August 3, 2023 03:19

github-actions bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: module (update) This PR changes an existing module in `nixos/` labels Aug 3, 2023

ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 1-10 labels Aug 3, 2023

Gerg-L mentioned this pull request Aug 3, 2023

Feature Request: boot.loader.systemd-boot.mirroredBoots #152155

Open

jian-lin requested review from ElvishJerricco and RaitoBezarius August 3, 2023 19:11

nixosTest testing

b37413a

Gerg-L force-pushed the systemd-boot branch from 885c938 to b37413a Compare August 6, 2023 01:13

Gerg-L marked this pull request as draft August 6, 2023 01:13

RaitoBezarius requested a review from nikstur August 6, 2023 01:32

nikstur requested a review from colemickens August 6, 2023 11:45

bjornfor reviewed Nov 21, 2023

View reviewed changes

ju1m reviewed Nov 22, 2023

View reviewed changes

Gerg-L closed this Dec 10, 2023

Gerg-L deleted the systemd-boot branch March 2, 2024 01:05

eclairevoyant mentioned this pull request Aug 28, 2024

Mirrored Boot option for systemd-boot #338017

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nixos/systemd-boot: Add mirroredBoots #246897

nixos/systemd-boot: Add mirroredBoots #246897

Gerg-L commented Aug 3, 2023 •

edited

Loading

0x4A6F commented Aug 3, 2023

RaitoBezarius commented Aug 3, 2023

Gerg-L commented Aug 3, 2023

Gerg-L commented Aug 3, 2023

Gerg-L commented Aug 5, 2023

Gerg-L commented Aug 6, 2023

nikstur commented Aug 6, 2023

Gerg-L commented Aug 6, 2023 •

edited

Loading

nikstur commented Aug 6, 2023

Gerg-L commented Aug 6, 2023

bjornfor commented Aug 6, 2023

nikstur commented Aug 6, 2023

Gerg-L commented Aug 6, 2023

bjornfor commented Aug 9, 2023

xaverdh commented Aug 9, 2023

Gerg-L commented Aug 9, 2023 •

edited

Loading

0x4A6F commented Aug 9, 2023

ElvishJerricco commented Aug 10, 2023

ElvishJerricco commented Aug 10, 2023

Gerg-L commented Aug 10, 2023 •

edited

Loading

nixos-discourse commented Nov 20, 2023

bjornfor Nov 21, 2023

ju1m Nov 22, 2023

ju1m Nov 22, 2023

ju1m Nov 22, 2023

Gerg-L commented Dec 10, 2023

nixos-discourse commented Sep 11, 2024

	boot.loader.grub.mirroredBoots = optionals (cfg.devices != [ ]) [
	{ path = "/boot"; inherit (cfg) devices; inherit (efi) efiSysMountPoint; }
	];

nixos/systemd-boot: Add mirroredBoots #246897

nixos/systemd-boot: Add mirroredBoots #246897

Conversation

Gerg-L commented Aug 3, 2023 • edited Loading

Description of changes

Things done

0x4A6F commented Aug 3, 2023

RaitoBezarius commented Aug 3, 2023

Gerg-L commented Aug 3, 2023

Gerg-L commented Aug 3, 2023

Gerg-L commented Aug 5, 2023

Gerg-L commented Aug 6, 2023

nikstur commented Aug 6, 2023

Gerg-L commented Aug 6, 2023 • edited Loading

nikstur commented Aug 6, 2023

Gerg-L commented Aug 6, 2023

bjornfor commented Aug 6, 2023

nikstur commented Aug 6, 2023

Gerg-L commented Aug 6, 2023

bjornfor commented Aug 9, 2023

xaverdh commented Aug 9, 2023

Gerg-L commented Aug 9, 2023 • edited Loading

0x4A6F commented Aug 9, 2023

ElvishJerricco commented Aug 10, 2023

ElvishJerricco commented Aug 10, 2023

Gerg-L commented Aug 10, 2023 • edited Loading

nixos-discourse commented Nov 20, 2023

bjornfor Nov 21, 2023

Choose a reason for hiding this comment

ju1m Nov 22, 2023

Choose a reason for hiding this comment

ju1m Nov 22, 2023

Choose a reason for hiding this comment

ju1m Nov 22, 2023

Choose a reason for hiding this comment

Gerg-L commented Dec 10, 2023

nixos-discourse commented Sep 11, 2024

Gerg-L commented Aug 3, 2023 •

edited

Loading

Gerg-L commented Aug 6, 2023 •

edited

Loading

Gerg-L commented Aug 9, 2023 •

edited

Loading

Gerg-L commented Aug 10, 2023 •

edited

Loading