install: Fetch bound images from host's /var/lib/containers

[cgwalters]

Instead of fetching images from the network at install time, require them to be present in the host's container storage (the same place we expect to find the bootc image).

There are numerous advantages to this, such as the basic fact that this helps one avoid re-pulling images at each install.

But more importantly it aligns with the needs of "hermetic builds" which want to pre-fetch content from the network into the target environment.

For example when generating a disk image with bootc-image-builder or so, a build tool can consistently preload all the input images (including the bootc base) into the container store, and know that the install process to generate the disk will use those images.

xref #715

[ckyrouac]

👍 one possible edge case I thought of is when the host machine’s image storage is out of sync with the repo. e.g.

host: quay.io/bootc/image:latest -> sha:foo
repo: quay.io/bootc/image:latest -> sha:bar

I think with the current code, the sha:foo image will be bound to the host. This kind of issue, where old code is accidentally deployed, tends to be pretty annoying to troubleshoot. Maybe we should do a pull of each image first to make sure it is up to date on the host before copying it over?

[cgwalters]

That can already happen today with the base bootc image, and I think sometimes that's actually what you want! If we always did a pull, it can create nondeterministic behavior where you're debugging something and then we push a new base image to the registry or whatever, and suddenly what you're debugging changes!

I think for this reason it's why podman run and podman build e.g. don't default to pulling.

Further, sometimes people may intentionally want to set this situation up - if for example you want to test OS upgrades, you very much want to deploy an older version, while still leaving the image references queued to pull updates (i.e. you don't want to explicitly deploy via @sha256).

Finally though, even more generally: we backed away hard from having bootc-image-builder do a pull...and my argument there was about authfiles: osbuild/bootc-image-builder#418 (comment)

And we definitely would have the authfile problem here too if we started doing distinct pulls at install time. Today for example, one could validly stick secrets in cloud-init metadata only, so the pull secret is only available at runtime. Of course we could describe to users how to wire up pull secrets into the install-time container (as we do for Anaconda) but we avoid that problem by not talking to a registry at all on our own at install time.

Ultimately anyone who wants to fetch new images can just run podman pull - and that way they only download it once too.

[ckyrouac]

makes sense, thanks for the explanation!