rework tmt to use pre-built container

[cgwalters]

OK so we have a lot of CI sprawl, across GHA and Testing Farm among others. That's not going to go away, but we need to keep chipping away at it.

Testing Farm/packit is really RPM oriented and less so containers, and especially less so bootc

We've been through a few iterations of bootc/tmt/testing farm iteration. A while ago we added the bootc provisioner plugin but that only works on host systems with virtualization available (if run in a cloud would require nested virt) and is specific to bootc-image-builder. (Of course we could teach it to make custom AMIs etc. but eww)

More recently we added https://github.com/bootc-dev/bootc/blob/main/tmt/tests/bootc-install-provision.sh which uses the install to-existing-root flow which simplifies things hugely from a testing perspective. And of course - it has the same "you don't need to build disk images" simplicity for users!

But the massive downside of the way we're doing things right now is we build from source for each test. (Also it's annoying to share code between this and the "main" Dockerfile, though we could figure out how to automate that)

What I really really really want is to be able to just test a bootc container image I have already built.

Wedging containers somehow into packit + testing-farm + tmt

The trifecta of packit + testing-farm + tmt is mostly logical, but it's all quite complicated and one can clearly see the different "gaps" between them technically/organizationally.

By far the biggest thing here is again the use case of "I want to run my local build in the target environment" - xref teemtee/tmt#1018

OK so here's my thought:

• Kill the bootc-install-provision.sh thing
• Change our tmt tests to require a container image reference in the environment
• Require use of https://github.com/cgwalters/cstor-dist for local testing

Anyone see issues with that?

More on the "packit -> testing-farm bridge"

So right now AFAIK basically our CI is doing "build an RPM in copr driven by packit" and then passing that off to testing farm and we end up sticking that RPM into a local container build run in each test.

What I'm thinking here is for pull request CI we could probably try either:

• Patch packit to learn how to build and push containers too
• Figure out how to insert custom functionality like this into the CI flow (maybe packit has some kind of hook we could use?) Basically, wait for the COPR build and then we take over build+push container to e.g. ghcr.io/bootc-dev/bootc-pr-12345-<digest> and passing that into testing farm?

Alternative 2: Drop packit+copr

Another alternative which actually has a fair amount of appeal is to just stop using packit - we just use the upstream podman build (which doesn't build an RPM today).

Instead we build a container image in the CI job however (probably GHA to start) and then just kick off testing-farm+tmt with that as a source.

Alternative 3: Try to go deep on using Konflux for upstream tests

This would have some notable advantages and disadvantages...

Sub-problem: tmt wants various dependencies

Due to how tmt works it also wants to inject itself into the target environment. Ideally in most cases we avoid this. But for things like tmt-reboot and such we could try using a sysext or so...

Related issues

• test: add e2e test #1359
• https://gitlab.com/testing-farm/general/-/merge_requests/14

[martinpitt]

@cgwalters I'm afraid I don't have a good solution to that. I've fought with this myself, without much success.

Just to make sure to cover all alternatives: Have you already ruled out building, deploying, and running the bootc container in GHA? Since /dev/kvm became available in workflows, that works really nicely, that's how I set up bootc testing in system-roles: bootc-buildah-qcow.sh uses bootc-image-builder to turn a buildah container (with the test subject) into a .qcow2 file, which then runs in qemu (that script is rather complicated and rather l-s-r specific, but I'm sure you know how to boot a qcow in qemu -- or I'm happy to give you lots of examples).

Pro: A lot fewer moving parts, it's amazingly fast, and more or less what you'd do locally for testing (in fact, build it first to be comfy for humans, and then deploy it to a workflow)

Con: You bind yourself to GitHub, as /dev/kvm access isn't widely available in other CI systems. https://cirrus-ci.com/ is a notable and well-working exception, or directly using GCE.

[cgwalters]

In the end we're going to have multiple CI systems; we just have to to support both (at least) GHA and Testing Farm/Konflux in the wider ecosystem.

This issue is mostly about the latter. With GHA it's easy for me to make it container centric. With packit+testing farm...less so (AFAIK).

Now what's also important to me here is that the flow at least natively supports being container-centric and makes it easy to run both locally and in CI. I guess this whole thing is also starting to touch on https://gitlab.com/fedora/bootc/tracker/-/issues/2

[henrywang]

Here's the workflow I'd like to try:

• build job:

1. build bootc binary in container and replace the bootc inside the bootc image
2. bootc install to-disk to generate disk image
3. archive disk file (skip this step if run locally), one image can be used by the test jobs in parallel

• test job:

1. get disk image from archive
2. boot disk image with qemu-system
3. run tmt test against this VM

[cgwalters]

build bootc binary in container and replace the bootc inside the bootc image

i.e. clone the project and run a default podman build right?

run tmt test against this VM

Yes though related to all of this...I think what would also help here is to use the tmt adjust to make the provisioning environment configurable or so? I think we still want to support running tests in e.g. cloud instances via the replace workflow and I believe we can do that by injecting stuff into the provision phase.