Filesystem-optimized implementations of impermanence

[schiffy91]

Thank you for your work on impermanence! I think it's an awesome idea and tool.

Because I/O-bound workloads are very important to me (and the I/O impact from bind mounts are non-trivial #248 ), I spent some time hacking away at a (highly experimental) impermanence-inspired derivative for my NixOS config that does not use bind mounts. Instead, it makes use of BTRFS snapshots: A clean snapshot is taken post-install, and, during every boot, each BTRFS subvolume is snapshotted and then reset to its appropriate post-install state. Any paths that were marked for persistence are then rsync'ed back in-place, thereby preserving ACLs, symlinks/hardlinks, etc without the need for a bind mount. Note: There is probably a more efficient way to sync the persisted files using BTRFS send/receive, which may become important for big files/folders persisted across boot – but rsync, for now, is sufficient for a prototype.

Anyway, the general idea is that impermanence-like solutions, in theory, can utilize the advanced capabilities of some newer filesystems to get around bind mounts or symlinks, at the expense of compatibility (i.e. this should also work well on ZFS, though would be inefficient on ext4 etc)

Is there any appetite in moving impermanence in this direction, where different filesystems could have different and more performant implementations that maximally utilize their capabilities instead of designing for the lowest common denominator? If there's any interest, I've attached my code here: https://github.com/schiffy91/nixos/blob/main/modules/system/immutability.nix, and I'd be happy to clean it up. The code, right now, is definitely rough around the edges – you'll find TODO's to make this operation "atomic-ish" and a ton of tracing, which helped me debug some edge cases. But it should hopefully be relatively legible to get an understanding of what a BTRFS implementation of impermanence could look like, and how each filesystem could reasonably have their own implementation of a standard interface (and default implementation).

I have a few options that hint at how this is relatively extensible with a potentially unified interface at https://github.com/schiffy91/nixos/blob/main/modules/settings.nix. It borrows heavily from your code; I don't think it'll have many surprises:

    settings.disk.subvolumes.volumes = mkSetting (listOf (submodule{ 
      options = { 
        name = mkSetting str null; 
        mountPoint = mkSetting str null; 
        mountOptions = mkSetting (listOf str) [ "compress=zstd" "noatime" ]; 
        neededForBoot = mkSetting bool true;
        resetOnBoot =  mkSetting bool false;
        flag = mkSetting (enum [ "none" "swap" "snapshots" "root"]) "none";
      };
    }))
    [
      { name = "@root"; mountPoint = "/"; flag = "root"; resetOnBoot = true; }
      { name = "@home"; mountPoint = "/home"; resetOnBoot = true; }
      { name = "@nix"; mountPoint = "/nix"; }
      { name = "@var"; mountPoint = "/var"; }
      { name = "@snapshots"; mountPoint = "/.snapshots"; flag = "snapshots"; }
      { name = "@swap"; mountPoint = "/.swap"; mountOptions = []; flag = "swap"; neededForBoot = false; }
    ];

    settings.disk.immutability.persist.paths = mkSetting (listOf str) [
      "/etc/nixos"
      "/etc/machine-id"
      "/etc/ssh"
      "/usr/bin/env"
      "/var/log"
      "/var/lib/bluetooth"
      "/var/lib/nixos"
      "/var/lib/systemd/coredump"
      "${config.settings.boot.pkiBundle}"
      "/home/${config.settings.user.admin.username}/.config/dconf/user"
       …
    ];

Of note, the script can be generalized to only touch sub volumes that have been marked for resetOnBoot. Many other options are there simply because I use the same options for disko as well (so it can be simplified significantly if designed for just impermanence). I've made some helpers my script uses to get these variables pretty easily, which are passed to my shell script as parameters, and could conceivably be used as part of the module to set things up in an extensible way, just requiring the end user to mark the volumes they want to reset on boot, the paths they want persist across boot, and the path of each subvolume's clean snapshots.

settings.disk.subvolumes.names.resetOnBoot = mkSetting str (lib.concatMapStringsSep " " (volume: volume.name) (lib.filter (volume: volume.resetOnBoot) config.settings.disk.subvolumes.volumes));
settings.disk.subvolumes.nameMountPointPairs.resetOnBoot = mkSetting str (lib.concatMapStringsSep " " (volume: "${volume.name}=${volume.mountPoint}") (lib.filter (volume: volume.resetOnBoot) config.settings.disk.subvolumes.volumes));

[DavidPesticcio]

Similarly for ZFS: https://grahamc.com/blog/erase-your-darlings/

[DADA30000]

why the snapshot part tho? It doesn't change anything, I think rsync part is only relevant here, but isn't it basically just copying?
Ah wait, I'm dumb, I see now, you need snapshots so that you could sync appropriate persistent data