Enable composefs + transient root

[cgwalters]

Let's enable this in our -dev images right now to try to put together our new recommended default experience.

[cgwalters]

This requires ostreedev/ostree#3170

[cgwalters]

And this is blocked by osbuild/bootc-image-builder#18 because that will (hopefully) fix the SELinux labeling for the underlying backing dirs.

(That said I may just try tomorrow to do a quick tactical patch for bib)

[cgwalters]

Just to emphasize the difference is that after this lands, the deployed host will also run more like podman run does. dnf install on the client (deployment) side will "work" - except changes made will be lost across a subsequent bootc upgrade - in the same way that pulling a new new container image in podman or Kube will also lose written data onto the overlay.

We're flipping around a big tradeoff here in that respect. I think it will make the system "feel" quite different from e.g. CoreOS/AtomicDesktops/etc. And we will also want to document how to turn off the transient overlay for those users who don't want it.

Related to this, really, most container images run in production deployments would be much better done with a readonly root. Some deployments go to some effort to try to use quotas to limit writes to the container overlayfs, when it would again just be better done by making it strictly readonly.

But, I think this alignment with the container defaults will help at a practical level.

[vrothberg]

LGTM

Can you add what you wrote to the docs? I find the description very helpful and believe it's worth preserving/sharing.

[cgwalters]

I've pushed the combined result of this PR plus dependencies to quay.io/cgwalters/ostest for ease of testing.

[jlebon]

Hmm, why not turn on state overlays instead for /opt and /usr/local (which I assume is the primary driver for this)? Since this is all still in flux and heavily being tested, it'll give us a good opportunity to proof it out. Would require actually implementing ostreedev/ostree-rs-ext#573 though.

[cgwalters]

I like the idea of state overlays but again, it breaks the model "this works how docker works".

[cgwalters]

We should definitely document though how to turn off the transient root in a derived build (requires dropping an override config and regenerating the initramfs), and at the same time doing state overlays.

Would require actually implementing ostreedev/ostree-rs-ext#573 though.

While doing it via RUN ostree container commit would be nicer, one can just RUN mv /opt/something /usr/lib/opt/something no?

[jlebon]

IMO, it's quite a safe assumption to make that users will expect packages that install in /opt to work the same as they currently do on traditional CentOS/RHEL.

Would require actually implementing ostreedev/ostree-rs-ext#573 though.

While doing it via RUN ostree container commit would be nicer, one can just RUN mv /opt/something /usr/lib/opt/something no?

Yeah, I think that'd work (but obviously also breaks the "works just like in traditional flows" abstraction).

[cgwalters]

IMO, it's quite a safe assumption to make that users will expect packages that install in /opt to work the same as they currently do on traditional CentOS/RHEL.

Can you elaborate on "work the same"? The main salient difference here is persistence of extra files across updates, is that what you're referring to?

[jlebon]

IMO, it's quite a safe assumption to make that users will expect packages that install in /opt to work the same as they currently do on traditional CentOS/RHEL.

Can you elaborate on "work the same"? The main salient difference here is persistence of extra files across updates, is that what you're referring to?

Right, application state across upgrades. The fact it applies only to applications that install to /opt and not globally also makes it harder to map to a model. Unless you're versed in OSTree/packaging internals, it's not obvious why /opt should be different. Do we want to expose users to this difference or try to make things Just Work like they do in any other context (application Dockerfile or traditional systems)?

In a way, state overlays matches better the "implicit VOLUME /var" we've pivoted to recently. E.g. nuking /var and rebooting would correctly nuke all application state. That's not the case for transient root.

[cgwalters]

Right, application state across upgrades. The fact it applies only to applications that install to /opt and not globally also makes it harder to map to a model.

If in this statement "it" = "transient root", that's not true. It applies equally to /usr/local or for that matter arbitrary new toplevels like /some-custom-app.

Or stated simply:

• /: transient overlay
• /etc: 3 way merge
• /var: Like VOLUME /var

The big difference between stateoverlays and transient root is that /opt is not a special case, it's just on /.

(In both of course we still have /home -> /var/home, even though actually many production cases would actually be happier with a fully transient /home)

Hmm the more I think about it the more perhaps having root.transient persist across reboots by default is wrong...it conflicts more with a secure boot scenario.

Do we want to expose users to this difference or try to make things Just Work like they do in any other context (application Dockerfile or traditional systems)?

I am confused because it sounds like you're arguing for transient root when you mention Dockerfile - in both transient root (host) and podman run we get an overlayfs that goes away when the container exits. Now of course a container build process i.e. RUN foo captures state - but there's no merge process with any "hidden" state.

state overlays don't exist in the default container ecosystem.

In a way, state overlays matches better the "implicit VOLUME /var" we've pivoted to recently. E.g. nuking /var and rebooting would correctly nuke all application state.

It's a bit more nuanced that that as docker behavior doesn't quite match the ostree one for /var; the content is only copied the very first time the volume is created. We don't quite have an equivalent of this today.

But yes, I am coming around to the idea that having the overlayfs persist by default across reboots was a bad idea.

We could probably in theory just reuse systemd.volatile=overlay but at the minor cost of a double overlayfs.

[jlebon]

Right, application state across upgrades. The fact it applies only to applications that install to /opt and not globally also makes it harder to map to a model.

If in this statement "it" = "transient root", that's not true. It applies equally to /usr/local or for that matter arbitrary new toplevels like /some-custom-app.

Right, but /opt is the big ticket item here. /usr/local doesn't really have the issue with "colocated state". I'm sure there are apps out there that create /some-custom-app against the FHS that we might have to provide guidance for (briefly: if it's pure code, that should work as is, if it's pure state, it'd work to e.g. make it a /var symlink/bind-mount, if it's both code and state, it'd need a state overlay). We could also transparently enable state overlays for any toplevel dir we detect like this if we wanted... Just another systemd unit template instantiation.

But again, everything else on the system doesn't behave this way (given that everything else is in /usr), so I think it's fair to still characterize it as special-casing. :)

Do we want to expose users to this difference or try to make things Just Work like they do in any other context (application Dockerfile or traditional systems)?

I am confused because it sounds like you're arguing for transient root when you mention Dockerfile - in both transient root (host) and podman run we get an overlayfs that goes away when the container exits. Now of course a container build process i.e. RUN foo captures state - but there's no merge process with any "hidden" state.

Let me rephrase this a different way: users will be installing various applications in their Dockerfile as part of creating a derived container image. Some of those applications will be in /usr and some in /opt. To require users to know there is a difference between those two classes of applications invites friction. They don't need to know this today whether using a traditional system or using an application container (in both cases, /usr and /opt behave like each other).

state overlays don't exist in the default container ecosystem.

State overlays isn't trying to model the container ecosystem. It's trying to make what works in traditional systems also work in image-based systems.

[cgwalters]

Right, but /opt is the big ticket item here. /usr/local doesn't really have the issue with "colocated state". I'm sure there are apps out there that create /some-custom-app against the FHS that we might have to provide guidance for

Yes, I recently discovered that the beaker project packages tests as RPMs that install to /mnt/test - and also xref internal link.

But also remember...it's not just about RPMs. Because the OCI world created a model where applications own their own filesystem, it's really common to just put binaries in e.g. /app or whatever and not care about namespacing. Some people will likely try to carry that type of thing forward even into bootable host container images.

Should we have a linting tool against this? Probably...

But again, everything else on the system doesn't behave this way (given that everything else is in /usr),

This is definitely a point of disagreement per above. It'd be nice if we lived in that world...

State overlays isn't trying to model the container ecosystem. It's trying to make what works in traditional systems also work in image-based systems.

Right, I agree. The ostree semantic for /etc is very much in that vein too.

We could also transparently enable state overlays for any toplevel dir we detect like this if we wanted... Just another systemd unit template instantiation.

I am just scared by that because it has a much higher level of magic. And by "magic" I mean "what precisely is the state of my files". The core tradeoff that we already covered is automatically having extra state/logs persist (good) versus having unintentional drift persist across changes.

The idea behind image-based updates is having the sha256 digest of the image you get from bootc status be meaningful. And any non-obvious persistent state undermines that. (I don't know what I was thinking with having the root.transient persist across reboots by default)

[cgwalters]

Another angle I'd take this to is that stateoverlays conflicts with a secureboot direction, where we really must strictly separate signed code from local persistent (unverified/unsigned) state like logfiles etc.

Training people to do hacks like RUN ln -s /var/log/someapp.log /opt/someapp/log to redirect from /opt to /var to explicitly persist state (as is needed with either composefs root or transient root) is better aligned with that than stateoverlays.

(And to complete this picture we should be mounting /var with noexec, etc.)

[jlebon]

But again, everything else on the system doesn't behave this way (given that everything else is in /usr),

This is definitely a point of disagreement per above. It'd be nice if we lived in that world...

I'm confused. Do you not agree that the great majority of software we ship in our images keep their code in /usr and their state in /var? State overlays is a way to make exceptions like /opt more like the rest of the system whereas transient root is introducing a new conceptual model that applies only to them. I'm arguing that the former is less surprising than the latter.

We could also transparently enable state overlays for any toplevel dir we detect like this if we wanted... Just another systemd unit template instantiation.

I am just scared by that because it has a much higher level of magic.

Right, not necessarily saying we should do that. Mostly food for thought.

The idea behind image-based updates is having the sha256 digest of the image you get from bootc status be meaningful. And any non-obvious persistent state undermines that. (I don't know what I was thinking with having the root.transient persist across reboots by default)

I'll just repeat again that state overlays is about making /opt behave like the rest of system. The digest doesn't cover any content in /var today either. But yeah, we probably should have an easy way to tell apart code from state in /opt (which actually is something that's not straightforward today even on traditional systems).

[cgwalters]

I'm confused. Do you not agree that the great majority of software we ship in our images keep their code in /usr and their state in /var?

For "our images" today - absolutely. But we're blasting open the doors for arbitrary 3rd party code and build processes generating derived images here...

transient root is introducing a new conceptual model that applies only to them.

We may need to do a realtime chat on this. I don't agree with "new conceptual model" because I am arguing it's how docker run worked since it was introduced. I am also unsure what precisely you mean here by "them".

The digest doesn't cover any content in /var today either.

Right. Bu that's how it works with podman/docker run and volumes too (OK I'm going to add VOLUME /var into our default base images now). And more broadly really, the overall rough consensus accelerated by systemd and ostree but definitely predating them back to the FHS is to have /var be this, so that isn't surprising.

[cgwalters]

OK I'm going to add VOLUME /var into our default base images now).

➡️ CentOS/centos-bootc#306

[cgwalters]

Jonathan and I had a realtime chat on this. One thing he wasn't aware of is that in the composefs case there is no special read-only bind mount over /usr - it behaves the same as the rest of /. This is one of the big reasons that e.g. dnf install etc. just work when on a transient root.

(And again to restate more generally, I think this aligns with the container model)

We agreed that there is a big user experience difference between transient root and stateoverlays. And the default really matters.

If we go with transient by default, we should definitely create another base image that disables transient root, and enables stateoverlay for /opt for example - just to make it even easier to play with.

Another option I guess is to just do composefs by default, and document (and test) both transient root and stateoverlays. (And hmm, I guess we should actually change bootc usroverlay to bootc rootfs enable-transient or so...)

[jlebon]

Jonathan and I had a realtime chat on this. One thing he wasn't aware of is that in the composefs case there is no special read-only bind mount over /usr - it behaves the same as the rest of /. This is one of the big reasons that e.g. dnf install etc. just work when on a transient root.

(And again to restate more generally, I think this aligns with the container model)

We agreed that there is a big user experience difference between transient root and stateoverlays. And the default really matters.

Indeed. Specifically, the trade off is essentially:

• state overlay: more hidden complexity, less user-facing complexity (/opt packages will appear to Just Work -- assuming Move content in /opt to /usr/lib/opt as part of ostree container commit if state overlays enabled ostreedev/ostree-rs-ext#573)
• transient rootfs: simpler upfront, more user-facing complexity (/opt packages with colocated state will have their state wiped on every reboot, unless users explicitly add e.g. symlinks from /opt/sub/dir to /var)

If we go with transient by default, we should definitely create another base image that disables transient root, and enables stateoverlay for /opt for example - just to make it even easier to play with.

Another option I guess is to just do composefs by default, and document (and test) both transient root and stateoverlays. (And hmm, I guess we should actually change bootc usroverlay to bootc rootfs enable-transient or so...)

I'm OK with either approach. I'm genuinely interested in what testers/users think of the two UXs. I mostly want the state overlay approach to be given fair consideration and get feedback. I'm less sure about supporting both longer term since it makes the story messier; I'm happy to drop the state overlay bits if it's judged to not add sufficient value.

[cgwalters]

OK actually since we did ostreedev/ostree@0cff65d this means we actually aren't blocked by osbuild/bootc-image-builder#149 anymore!

[cgwalters]

Now proposing this in the main image in CentOS/centos-bootc#356