Support multiple VMs of the same container image

[germag]

Currently, there is a 1:1 map between the VM in the cache and the OCI image.

I believe there are valid use cases for wanting to run more than one VM of the same OCI images. The point is, do these use cases fall within the scope of podman-bootc?, What would it look like from the UI POV?, for instance, the user needs to request a new VM:

podman bootc run --new-vm <imagename>

or every run creates a new VM if there is an existent VM already

How that will work when we need to rebuild the disk image, if for instance the disk size changes:

1) podman-bootc run --disk-size 20G image0
...
2) podman-bootc run --disk-size 25G image0

2) should create a new VM?, or only if 1) is still running. Or we replace the previous one unless --new-vm?, or by default is a new vm unless --replace-vm?

If we have more than one VM of the same OCI image, which of those we start on run, should we need to add --vm <vm_id>?

If we don't want to do this "automagically", we will need to manage the VMs as persistent objects with its own IDs.
I want to have this discussion before starting the work on the cache code, because it changes pretty much everything.

[germag]

@cgwalters @ckyrouac thoughts?

[cgwalters]

The question of scope is a tricky one. I think on Linux, we should at least have it feel like a "smooth transition" to go from the simple use cases embodied here to e.g. virt-install - and that leads us down the topic of whether virt-install itself reuses parts of this project or not. This also intersects with discussion in #28

I'd lean towards at least on Linux having something like podman bootc create-libvirt instead of run which would take a name for the instance, and after that hand off its entire lifecycle to libvirt; i.e. it wouldn't show up in podman-bootc list, etc.

[ckyrouac]

I'd lean towards at least on Linux having something like podman bootc create-libvirt instead of run which would take a name for the instance, and after that hand off its entire lifecycle to libvirt; i.e. it wouldn't show up in podman-bootc list, etc.

I like this general flow. I definitely prefer to avoid duplicating existing VM management APIs. IMO keeping podman-bootc scoped to the original idea of enabling rapid testing/development of bootc images would be best. We could use podman-bootc to direct users to more advanced workflows, e.g. printing a message after podman-bootc run about next steps like building an image with image builder, deploying to cloud, etc. An SDK of some form that includes all the related tools might help with this too. If there's enough interest in simplifying those advanced workflows then it might make more sense to include those in podman-bootc.

[cgwalters]

And on macOS it'd make a lot of sense to support an "export to lima" flow.

[germag]

The question of scope is a tricky one. I think on Linux, we should at least have it feel like a "smooth transition" to go from the simple use cases embodied here to e.g. virt-install - and that leads us down the topic of whether virt-install itself reuses parts of this project or not. This also intersects with discussion in #28

I like this general flow. I definitely prefer to avoid duplicating existing VM management APIs. IMO keeping podman-bootc scoped to the original idea of enabling rapid testing/development of bootc images would be best. We could use podman-bootc to direct users to more advanced workflows, e.g. printing a message after podman-bootc run about next steps like building an image with image builder, deploying to cloud, etc. An SDK of some form that includes all the related tools might help with this too. If there's enough interest in simplifying those advanced workflows then it might make more sense to include those in podman-bootc.

From a sysadmin POV, I can think of use cases where the simple test requires more than one VM.

I'd lean towards at least on Linux having something like podman bootc create-libvirt instead of run which would take a name for the instance, and after that hand off its entire lifecycle to libvirt; i.e. it wouldn't show up in podman-bootc list, etc.

I'm not sure I like this approach, for instance we will need to know where to store the disk image (we cannot keep it in the cache).

I think it's simpler to teach virt-install to create bootc images and/or have a podman-bootc export/save/copy command that can copy the disk image outside the cache, then the user can use that disk image with virt-install or lima or any other VMM (we should also have an inspect command), so we don't need to support any particular one

[ckyrouac]

From a sysadmin POV, I can think of use cases where the simple test requires more than one VM.

There's definitely use cases for testing multiple VMs. I'm not sure there is much value in creating a contrived test environment via podman-bootc though. I imagine most (if not all) users who want to test interoperability will have a testing environment setup, e.g. a cloud env, test lab, etc. Those environments usually have unique networking, storage, etc. requirements that are the more interesting thing to test. We can't really cover all these nuances via podman-bootc without essentially duplicating existing tools for those environments. They also probably have existing machines and services that need to be integration tested with the new bootc image.

podman-bootc export/save/copy command that can copy the disk image outside the cache, then the user can use that disk image with virt-install or lima or any other VMM (we should also have an inspect command), so we don't need to support any particular one

This would essentially act as a simplified version of bootc image builder, right?

In my mind, podman-bootc is part of a suite of tools used for developing a bootc image. So, a typical workflow would be something like:

1. Create a repo with a Containerfile and supporting files (maybe bootstrapped from a template/example)
2. podman build . -t my-image
3. podman-bootc run my-image
4. initial validation the image is running
5. modify the Containerfile by adding packages, custom code, etc.
6. podman-bootc run my-image
7. validate changes, repeat 5-6 until it's working right
8. build the final image via bootc image builder
9. deploy the image to test environment and smoke test with other existing services
10. add the bootc image builder disk creation to a CI environment, if needed
11. deploy to prod env

[germag]

From a sysadmin POV, I can think of use cases where the simple test requires more than one VM.

There's definitely use cases for testing multiple VMs. I'm not sure there is much value in creating a contrived test environment via podman-bootc though. I imagine most (if not all) users who want to test interoperability will have a testing environment setup, e.g. a cloud env, test lab, etc. Those environments usually have unique networking, storage, etc. requirements that are the more interesting thing to test. We can't really cover all these nuances via podman-bootc without essentially duplicating existing tools for those environments. They also probably have existing machines and services that need to be integration tested with the new bootc image.

I'm not sure adding multiple VMs is too contrived, if we design the cache to supports them (I'm not talking about supporting complex scenarios).

I'm also not advocating to support multiple VMs, I just want to have this discussion if is it worth it or not, before embarking on changing the cache code. Because if we want to add them later will be difficult.

podman-bootc export/save/copy command that can copy the disk image outside the cache, then the user can use that disk image with virt-install or lima or any other VMM (we should also have an inspect command), so we don't need to support any particular one

This would essentially act as a simplified version of bootc image builder, right?

kind of, yes, but I think is simpler and more flexible than supporting create-libvirt|lima|UTM|etc. Can bootc image builder use a local image now, or it still needs to pull the image from a register?

In my mind, podman-bootc is part of a suite of tools used for developing a bootc image. So, a typical workflow would be something like:

1. Create a repo with a Containerfile and supporting files (maybe bootstrapped from a template/example)

2. `podman build . -t my-image`

3. `podman-bootc run my-image`

4. initial validation the image is running

5. modify the Containerfile by adding packages, custom code, etc.

6. `podman-bootc run my-image`

7. validate changes, repeat 5-6 until it's working right

8. build the final image via bootc image builder

9. deploy the image to test environment and smoke test with other existing services

10. add the bootc image builder disk creation to a CI environment, if needed

11. deploy to prod env

That was also my idea when a started this project, and I still think the same, but I cannot deny that steps 4 and 7 are not well-defined, for instance, to test if a master-slave dns server works we need to 2 VMs (and that is quite simple test). The issue I see is we have two main type of users: developers and sysadmins/devops.

[ckyrouac]

but I cannot deny that steps 4 and 7 are not well-defined

To me, these steps are just basic validation of the machine's state, e.g. did the machine boot? are the correct services running? are there any errors in the journal? are all the custom files in place? etc.

[germag]

but I cannot deny that steps 4 and 7 are not well-defined

To me, these steps are just basic validation of the machine's state, e.g. did the machine boot? are the correct services running? are there any errors in the journal? are all the custom files in place? etc.

Makes sense to me

[rhatdan]

I am fine with podman bootc extract --output /tmp/myimage.qcow2 quay.io/rhatdan/myimage

But we should not go any further then that. Otherwise you are creating something to compete with bootc-image-builder.

[germag]

I am fine with podman bootc extract --output /tmp/myimage.qcow2 quay.io/rhatdan/myimage

But we should not go any further then that. Otherwise you are creating something to compete with bootc-image-builder.

Yep, I was thinking something in the line of podman save so the workflow would be something like:

podman bootc run IMAGE
... 
podman bootc extract --output /tmp/myimage.qcow2 IMAGE