systemd-remount-fs.service fail with composefs enabled

[karuboniru]

• Enable composefs by adding following lines to ostree-prepare-root.conf:

[composefs]
enabled=yes

• Apply image and reboot
• Check systemctl status systemd-remount-fs.service

Result:

2月 22 13:28:48 karubook systemd-remount-fs[19855]: mount: /: mount point not mounted or bad option.
2月 22 13:28:48 karubook systemd-remount-fs[19855]:        dmesg(1) may have more information after failed mount system call.
2月 22 13:28:48 karubook systemd-remount-fs[19854]: /usr/bin/mount for / exited with exit status 32.
2月 22 13:28:48 karubook systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE

---

This might be a systemd bug that tried to re-mount the overlay on / type overlay (ro,relatime,seclabel,lowerdir=/run/ostree/.private/cfsroot-lower::/sysroot/ostree/repo/objects,redirect_dir=on,metacopy=on). If you can comfirm this I can file an another issue on systemd repo.

[karuboniru]

Cleaning the / entry in /etc/fstab can workaround this, while this introduced another issue where during shutdown there is a [Failed] Failed unmounting sysroot.mount - /sysroot without any information being given in journal.

[cgwalters]

What's in your /etc/fstab?

[karuboniru]

# UUID=1eef9f42-40e3-4bd8-ae20-e9f2325f8b52 /                     btrfs   subvol=root,compress=zstd:1,noatime,user_subvol_rm_allowed,ro 0 0
UUID=f35b55c6-f8c3-4279-be97-8811abda483a /boot                   ext4    defaults        1 2
UUID=6907-17CA          /boot/efi               vfat    umask=0077,shortname=winnt 0 2
UUID=1eef9f42-40e3-4bd8-ae20-e9f2325f8b52 /var                    btrfs   subvol=var,compress=zstd:1,noatime,user_subvol_rm_allowed 0 0
UUID=1eef9f42-40e3-4bd8-ae20-e9f2325f8b52 /var/home               btrfs   subvol=home,user_subvol_rm_allowed,noatime     0 0

I understand that the remount failed due to / no longer being a "naked btrfs mount` but now an overlayfs. So I commented it out.

[cgwalters]

The general fix here is to use rw in your kernel commandline. Hmm, actually you have ro in your mount options...is that intentional? What's in your /proc/cmdline? Was your rootfs being mounted rw before? You have writable subvolumes, but still...

[karuboniru]

actually you have ro in your mount options...is that intentional

Sorry, that should be added during the process when I am trying to workaround the issue myself. I first tried to add the ro and it did not work then I commented the whole line out.

The cmdline is

BOOT_IMAGE=(hd0,gpt2)/ostree/fedora-0ce8c3b357a4e48a8c5b1617d75fbdb00219f87679fc273acb00dd831cddd07d/vmlinuz-6.8.0-0.rc5.41.fc40.x86_64 rhgb quiet root=UUID=1eef9f42-40e3-4bd8-ae20-e9f2325f8b52 rootflags=subvol=root rw ostree=/ostree/boot.0/fedora/0ce8c3b357a4e48a8c5b1617d75fbdb00219f87679fc273acb00dd831cddd07d/0

[cgwalters]

I think we'll need to patch systemd for this most likely; there's a bit of a conflict here started from the basic fact that composefs is inherently an overlayfs on top of the underlying real root, and systemd-remount-fs is unaware of this.

Let's take as a prerequisite that we have an elegant way for external tooling to "peel" the overlayfs and find the backing filesystem - singular. In theory a composefs (as it's an overlayfs) can span multiple filesystems, but I struggle to think of a realistic use case for that - at least for the root filesystem.

The systemd volatile root logic has an ad-hoc symlink it creates...would be nicer to be able to attach this as in-band metadata on the mount somehow. But I guess for our use case, similar to the systemd one we're just handling the single root filesystem so a single well-known symlink is probably OK for now.

We then have a tension around the writability of the underlying real root vs the composefs. This issue relates to #3177 a lot.

I'd still take the basic stance that it just makes sense to mount the real root writable (if that's the intent) from the initramfs from the start.

Probably the most practical change is for systemd-remount-fs to gain some logic for detecting that the root it sees is an overlayfs, and if the backing fileystem matches that defined in /etc/fstab, it can handle remounting that instead. And in the common/expected case, the writability and mount options should match and it should be a no-op.

[cgwalters]

Some code for this in containers/bootc#417 - but as I said there it's really papering over work we need to do probably in both anaconda and systemd.