Add new registry section

[schaefi]

Allow to specify references to OCI containers in the image description like in the following example:

<registry source="registry.example.com">
    <container name="some" tag="latest" path="some/path" use_with="podman"/>
    <container name="some_other" use_with="docker" fetch_only="true"/>
</registry>

During the kiwi process the containers are fetched into a temporary location and a systemd service is configured to one time load the containers into the local registry at first boot of the system.

This Fixes #2663

[schaefi]

There are two aspects that are not covered by this PR

1. The container tool chain needs to be part of the image. Meaning skopeo and the selected backend e.g podman needs to be explicitly installed via the package selection. There is no automation that would pull them in
2. Loading the containers into the storage backend requires space to be available in the image. There is no extra space addition done for this loading process because it cannot be predicted as it depends on the storage backend. Thus it's also a user task to give the image enough space such that the on-boot loading operation can succeed

[Conan-Kudo]

I'm not sure this is the model we want. Why is the registry the top-level element with containers inside? Why isn't this part of sources, and then we can add containers and flatpaks to the same section packages are set? Also, why do we wait until firstboot to load them into container storage?

[schaefi]

I'm answering in reverse order:

Also, why do we wait until firstboot to load them into container storage?

The reason for this is because loading a container archive into the storage area of the container backend requires the configured backend to be up and active. For example in podman your storage backend could be btrfs based and living in a snapshot which would not be able to be created/managed while we are building the offline image root tree. For e.g docker it's required that the docker daemon runs to actually load a container into the storage space, same for containerd... etc etc etc. It simply comes with too many restrictions when we want to load the container into the storage backend while it is not up and running. I don't want this limitation and therefore the model is fetch offline, load online. The price to pay for this model is storage space but that can be handled.

Why isn't this part of sources, and then we can add containers and flatpaks to the same section packages are set?

The reason why I used a new toplevel element for this is because our sources model is based on repository followed by packages and a package manager that knows how to resolve using all repositories. When doing the same with containers we have no "container package manager" and have to tell in the image description which container should be fetched from which registry. Because of that I added a new section named registry and you need to bind the containers that you want to fetch from this registry via container sub-sections to that registry. This concept could grow with other data like flatpacks. In general with all data that is delivered through a registry

I'm open for other definition semantics but would like to ask for an example as you see it.

[rjschwei]

The "request changes" setting is for the spelling, all other comments are suggestions to consider, primarily from my perspective as a user of the feature.

[Conan-Kudo]

I'd also like @travier to look at this and give feedback because I think we can kill two birds with one stone here and figure out the OCI Flatpak stuff that is needed for Fedora too.

[schaefi]

I'd also like @travier to look at this and give feedback because I think we can kill two birds with one stone here and figure out the OCI Flatpak stuff that is needed for Fedora too.

ok, the final definition section looks like the following example

<registry source="registry.opensuse.org">
    <containers backend="docker" path="home/marcus.schaefer/delta_containers/containers_tw">
        <container name="joe"/>
    </containers>
    <containers backend="podman" path="home/marcus.schaefer/delta_containers/containers_tw">
        <container name="mc"/>
        <container name="lynx"/>
    </containers>
</registry>

If we need to add more to handle flatpacks let me know

[Conan-Kudo]

Is there a reason we can't add registry sources as repositories with oci type and then be able to reference them using the alias attribute for containers? Also, why should registry be mandatory here, since you can pull container images with fully qualified paths (e.g. registry.opensuse.org/git)? I would prefer we more closely mimic what people do on the commandline here and require fully qualified paths instead.

[schaefi]

Is there a reason we can't add registry sources as repositories with oci type and then be able to reference them using the alias attribute for containers? Also, why should registry be mandatory here, since you can pull container images with fully qualified paths (e.g. registry.opensuse.org/git)? I would prefer we more closely mimic what people do on the commandline here and require fully qualified paths instead.

The reason is that a registry is not a repository in the meaning of kiwi. I asked you to make an example how you would express it. So I believe you had something like this in mind

<repository type="oci">
    <source path="registry.opensuse.org/home/marcus.schaefer/delta_containers/containers_tw"/>
</repository>

<packages>
    <container name="joe"/>
    <container name="foo"/>
</packages>

If the container foo should come from another registry how would you reference that ? There is no container package manager that would work on a number of repository entries of type oci and I personally find this completely confusing.

In addition if we intermingle the registry handling with the repositories we have to touch all package manager backends to ignore type="oci".

I'm sorry I see this as two different things. As for the commandline, we would need to add new commandline switches that says --add-registry, --set-registry if need be but also not intermix this with the existing --add-repo, --set-repo options

[schaefi]

next we would have to add a backend information (podman/docker/containerd, etc...) to the repository element which is an information that does not exist for repositories but for registries. So we would add attributes to an element which is pointless for package repositories. And vice versa many repository attributes are pointless for registries. That doesn't sound like a good semantic to me

[schaefi]

Also there might be additional changes on top of this feature which we currently don't know about. I would like to keep the repository and package management code free from any code change related to registry handling

[Conan-Kudo]

So my confusion about the registry section is that it's reasonable to do podman pull registry.opensuse.org/joe:latest. In this sense, we don't need a registry tag at all, since it's implied in the name of the container being fetched.

Now if we want to create some kind of construction mechanism instead of just passing it straight to the container tools, then we also need to handle the tag too, which I don't see a semantic for. EDIT: I see we have that already.

And also, how do we handle when the "registry" tag is used multiple times in different snippets? Is it merged? Does it conflict? What do we do here? And what about non-OCI things like Incus? And non-OCI Flatpaks?

[schaefi]

So my confusion about the registry section is that it's reasonable to do podman pull registry.opensuse.org/joe:latest. In this sense, we don't need a registry tag at all, since it's implied in the name of the container being fetched.

Now if we want to create some kind of construction mechanism instead of just passing it straight to the container tools, then we also need to handle the tag too, which I don't see a semantic for.

And also, how do we handle when the "registry" tag is used multiple times in different snippets?

I can't follow you

<registry source="registry.opensuse.org">
    <containers backend="podman">
        <container name="joe"/>
        <container name="foo" tag="some"/>
        <container name="bar" fetch_only="true"/>
    </containers>
    <containers backend="docker" path=take/that>
        <container name="some"/>
    </containers>
</registry>
<registry source="registry.fedora.org">
    <containers backend="podman">
        <container name="xxx"/>
    </containers>
</registry>

translates to:

• podman pull registry.opensuse.org/joe:latest
• podman pull registry.opensuse.org/foo:some
• only fetching of registry.opensuse.org/bar:latest no loading into any backend
• docker pull registry.opensuse.org/take/that/some:latest
• podman pull registry.fedora.org/xxx:latest

Where do you see issues ? and as I said I consider this very different than the repository handling and don't want to mix it

[Conan-Kudo]

What happens when we have two registry="registry.fedoraproject.org" tags? What behavior do we have there? Also, are registry and container tags able to be arched like packages? Not all containers are available on all arches. What about profiles?

But about the registry thing, why wouldn't we just do:

    <containers backend="podman">
        <container name="registry.opensuse.org/joe"/>
        <container name="registry.opensuse.org/foo" tag="some"/>
        <container name="registry.opensuse.org/bar" fetch_only="true"/>
    </containers>
    <containers backend="docker" path="take/that">
        <container name="docker.io/library/some"/>
    </containers>

My point is I don't think we need registry per se.

[rjschwei]

So my confusion about the registry section is that it's reasonable to do podman pull registry.opensuse.org/joe:latest. In this sense, we don't need a registry tag at all, since it's implied in the name of the container being fetched.

Correct, but it gets a bit tiresome when there is more than 1 container and one has to type or generate the same registry path over an over again. Having a collector element that specifies the registry from which to pull a bunch of containers is a convenience feature for the user creting the kiwi description.

Now if we want to create some kind of construction mechanism instead of just passing it straight to the container tools, then we also need to handle the tag too, which I don't see a semantic for. EDIT: I see we have that already.

And also, how do we handle when the "registry" tag is used multiple times in different snippets? Is it merged? Does it conflict? What do we do here?

I do not see a reason why merging logic should be implemented or the should be any conflict.

<registry source="registry.opensuse.org">
   <containers backend="podman">
      <container name=foo"/>
  </containers>
</registry>
<registry source="registry.opensuse.org">
   <containers backend="docker">
      <container name=foo"/>
  </containers>
</registry>
"""
Is perfectly fine IMHO. 

> And what about non-OCI things like Incus? And non-OCI Flatpaks?

Should this really be handled with the same mechanism?

[Conan-Kudo]

@rjschwei I don't want us to be trapped with awkward semantics just because we didn't think of it like we are with systemdisk and some others. I'm thinking about all the other things that matter with supporting "containers" in kiwi that have or will come up.

[Conan-Kudo]

And if we want to have the "convenience" of a registry tag, then I'd rather it be inside the containers top level tag rather than the containers tag being nested inside the registry tag.

[rjschwei]

What happens when we have two registry="registry.fedoraproject.org" tags? What behavior do we have there? Also, are registry and container tags able to be arched like packages? Not all containers are available on all arches. What about profiles?

But about the registry thing, why wouldn't we just do:

    <containers backend="podman">
        <container name="registry.opensuse.org/joe"/>
        <container name="registry.opensuse.org/foo" tag="some"/>
        <container name="registry.opensuse.org/bar" fetch_only="true"/>
    </containers>
    <containers backend="docker" path="take/that">
        <container name="docker.io/library/some"/>
    </containers>

My point is I don't think we need registry per se.

Why do you want to type, or make the user type registry.opensuse.org possibly 100s of times?

[Conan-Kudo]

Now that I think about it more, why wouldn't we just define the registry as an attribute of containers?

    <containers backend="podman" source="registry.opensuse.org">
        <container name="joe"/>
        <container name="foo" tag="some"/>
        <container name="bar" fetch_only="true"/>
    </containers>
    <containers backend="docker" source="docker.io/library" path="take/that">
        <container name="some"/>
    </containers>

[Conan-Kudo]

And what about non-OCI things like Incus? And non-OCI Flatpaks?

Should this really be handled with the same mechanism?

Yes. We are not in the business of introducing custom things for every single thing. So maximizing reuse of our tags and attributes is beneficial.

[rjschwei]

And if we want to have the "convenience" of a registry tag, then I'd rather it be inside the containers top level tag rather than the containers tag being nested inside the registry tag.

I have no strong preference between

<containers backend="podman">
    <registry source="registry.opensuse.org">
        <container name="foo"/>
        <container name="bar"/>
    </registry>
    <registry source="registry.fedoraproject.org">
       <container name="blue"/>
       <container name="green"/>
    </registry>
</containers>

which is what I think your proposing or the order in the PR. That said, it does look at bit odd to have the container element be a child of registry rather than containers but that's fine by me.

[rjschwei]

Now that I think about it more, why wouldn't we just define the registry as an attribute of containers?

    <containers backend="podman" source="registry.opensuse.org">
        <container name="joe"/>
        <container name="foo" tag="some"/>
        <container name="bar" fetch_only="true"/>
    </containers>
    <containers backend="docker" source="docker.io/library" path="take/that">
        <container name="some"/>
    </containers>

Shouldn't the path go wit the container? Then I can pick up different containers that are in different locations of the same registry.

[schaefi]

Now that I think about it more, why wouldn't we just define the registry as an attribute of containers?

    <containers backend="podman" source="registry.opensuse.org">
        <container name="joe"/>
        <container name="foo" tag="some"/>
        <container name="bar" fetch_only="true"/>
    </containers>
    <containers backend="docker" source="docker.io/library" path="take/that">
        <container name="some"/>
    </containers>

I would be fine with this change

[Conan-Kudo]

Shouldn't the path go wit the container? Then I can pick up different containers that are in different locations of the same registry.

You would be able to define multiple containers sections with different registries, and then set them up as appropriate. This is just more compact than the current representation in the PR.

[schaefi]

I'm fine with the more compact definition. I'll change it ... after some rest, today evening or by tomorrow when I'm back at work

[Conan-Kudo]

We also need the containers to be able to be profile tagged like others too. That way things like Fedora's setup for their kiwi descriptions will be able to work with description fragments. And containers are arch-specific, so individual containers need to be able to be tagged with arch as needed.

[schaefi]

Note that individual container lines need to support the arch attribute too, since that can be per container.

like with the packages, yes I will add that

[schaefi]

Hmm, looking again at Robert's suggestion, that also moves the backend definition as an attribute to the containers element, which invalidates my idea of nesting the containers into a backend definition. In this case also the path element can stay in the container element because none of the advantages for having a backend element exists anymore. Actually this is very similar to what I started with, but I finally would like to move forward here and as it seems you both can live with that I'm going to add this suggestion

[travier]

Is there any specific reason that we are not using the URL format from https://github.com/containers/image/blob/main/docs/containers-transports.5.md for referencing the container images? Not all containers come from registries so making this a required element feels weird.

What's the reason behind the podman/docker option? Both should handle the same container images fine.
Edit: Ah, it's to specify how to generate the systemd unit that will load the container into the storage? It feels like that should be part of something else and not managed by kiwi as this is modifying the content of the final system.

[travier]

Overall, I like this option. I think I should be able to use it to make things simpler for the ostree/bootable container path.

[travier]

Thinking about this more, I think we should not focus on how the import is made but more on how the container image is stored in the image.

The idea would be to specify where the image should go and under which format and then it would be up to the user doing the image to handle how to import it. This would let us pre-setup the containers for the backends that support it.

Example:

<containers>
     <!-- Pull the image from a registry and store it as an OCI archive at the given path -->
     <container source="docker://docker-reference" dest="oci:path[:reference]"/>
     <!-- Pull the image directly from the local container storage and store it in the image destination storage -->
     <container source="containers-storage:[[storage-specifier]]{image-id|docker-reference[@image-id]}" dest="containers-storage:[[storage-specifier]]{image-id|docker-reference[@image-id]}"/>
     <!-- Pull the image from a local OCI archive and store it at the given path -->
     <container source="oci:path[:reference]" dest="oci:path[:reference]"/>
</containers>

[schaefi]

In reply to @travier idea

• a direct pull of the container into the local container storage backend is possible only in a very limited way when building the image because the storage backend is offline. For podman I could make this work only with the vfs storage backend driver and in many other cases I stumbled over issues like: conflicts with overlay because podman thinks the real-root is not the system, conflicts with podman which switches to fuse-overlayfs when it should use overlay, conflicts with docker which requires dockerd to run for importing containers, conflicts with nested workloads when you want to build an image in a container and you run nested imports, etc etc etc.... the list is endless. That's the reason why we don't support the direct import and offered a two stage system where we store the container archive locally and load it into the container storage backend on first boot via a systemd service

• With the support for containers that we store intermediate in the image we made two static settings:

  1. The transport protocol to store the container image is oci-archive and connected to the configured backend name. I don't see a benefit to let a user make this decision because this is not data the user handles, we handle it for the purpose of loading the container to the local registry. So the tooling to use and the transport protocol is encoded in kiwi according to the selected backend. We currently support docker and podman and for both we copy docker:// to oci-archive:target:source
  2. The location to store those images is set to /var/tmp/kiwi_containers. It's intentionally not configurable at the moment as it is considered a temporary space to store the container archives until they get loaded into the container storage backend. The systemd service which we write deletes the container archive as soon as it is loaded into the local storage unless the fetch_only attribute is set

We can have a conversation if it would be beneficial to add additional attributes to influence some of the fixed settings, but I would do this on real demand in followup pull requests.

Thanks

[schaefi]

Is there any specific reason that we are not using the URL format from https://github.com/containers/image/blob/main/docs/containers-transports.5.md for referencing the container images? Not all containers come from registries so making this a required element feels weird.

I can't follow you on this. We use the official supported transport URL. The final URL is constructed out of the
source, name, tag and path information. source specifies an endpoint and how that endpoint should be reached is
determined by the specified backend

What's the reason behind the podman/docker option? Both should handle the same container images fine. Edit: Ah, it's to specify how to generate the systemd unit that will load the container into the storage? It feels like that should be part of something else and not managed by kiwi as this is modifying the content of the final system.

With the backend name we determine several settings, how to reach the endpoint, which tools to use, how to load the
image. The backend name gives us also the opportunity to evaluate the given container data also for other containers
types, e.g flatpacks, or whatever else comes in the future

It feels like that should be part of something else and not managed by kiwi as this is modifying the content of the final system.

I'm afraid as I tried to explain that a direct import while building is very limited and is only causing trouble I only saw the way via a one-shot firstboot service on boot. If you can be more explicit what you have in mind by something else we could adapt the concept :)

[travier]

I'm afraid as I tried to explain that a direct import while building is very limited and is only causing trouble I only saw the way via a one-shot firstboot service on boot. If you can be more explicit what you have in mind by something else we could adapt the concept :)

What I meant by something else is that the user would write this unit and control its behavior. But I can understand how that could fit into the current behavior. Are there any other occurrences in Kiwi where it directly writes systemd units to the system?

Is there any specific reason that we are not using the URL format from containers/image@main/docs/containers-transports.5.md for referencing the container images? Not all containers come from registries so making this a required element feels weird.

I can't follow you on this. We use the official supported transport URL. The final URL is constructed out of the source, name, tag and path information. source specifies an endpoint and how that endpoint should be reached is determined by the specified backend

We currently support docker and podman and for both we copy docker:// to oci-archive:target:source

So if I understand correctly, this will always generate a "docker://" URL and copy to an OCI archive.

If you can not specify that you want to take it from the current local storage for example then this either means pulling it every time (to make sure it is the latest version) or caching it locally (and then needing a way to force a refresh).

This is part of why I suggested that we don't split those and use the full transport URL (at lest for the source) so that the user can decide and manage that on their own.

[travier]

To clarify, I'm only sharing thoughts here and you should feel free to ignore anything I say :). I don't think we'll be able to use that directly for the bootable container cases in this current form.

[schaefi]

What I meant by something else is that the user would write this unit and control its behavior. But I can understand how that could fit into the current behavior. Are there any other occurrences in Kiwi where it directly writes systemd units to the system?

Nope this is the first time that we do this and I also think it smells a bit, but as I said I don't see another way except for letting the user alone with it. btw if you specify all containers with fetch_only="true" we only fetch the containers and do not create any systemd unit

[schaefi]

So if I understand correctly, this will always generate a "docker://" URL and copy to an OCI archive.

If the backend is set to either backend="podman" or backend="docker" yes

[schaefi]

If you can not specify that you want to take it from the current local storage for example then this either means pulling it every time (to make sure it is the latest version) or caching it locally (and then needing a way to force a refresh).

We currently fetch the container at build time of the image and store it there. On first boot it gets loaded. Assuming you build the image on a Monday and deploy it on a Friday, the container could have been updated in the registry yes, but the same applies to packages taken from a a repository.

I was not quite clear what you mean by specify that you want to take it from the current local storage ? which local storage ? the one of the build host you build the image on ?

[schaefi]

To clarify, I'm only sharing thoughts here and you should feel free to ignore anything I say :). I don't think we'll be able to use that directly for the bootable container cases in this current form.

I take this serious as it covers use cases which are the important part. A feature of no use is useless right :) But I agree this is not working towards the bootc idea. With the containers element we want to make containers available to the local storage of the image such that you can pre-populate a container storage backend with container images fetched at build time of the image from a trusted source

[Conan-Kudo]

We still need it even for bootc because it's entirely reasonable to expect a base image seeded with any and all content in an offline context. From my perspective, bootc is just saving things as an image layer that's pushed and pulled through OCI tooling. What is in the image layer is still much more like what we do with any other appliance image build.

[Conan-Kudo]

But I did not think about the fact that there's multiple input source types supported by podman and docker. Since the source= attribute is optional, we should be able to allow fully qualified references when it's not set. But if it's used, we probably should also add something to specify the protocol for the source. Or just say it needs to be specified in source?

[schaefi]

But I did not think about the fact that there's multiple input source types supported by podman and docker. Since the source= attribute is optional, we should be able to allow fully qualified references when it's not set. But if it's used, we probably should also add something to specify the protocol for the source. Or just say it needs to be specified in source?

source is not optional, you have to give an endpoint to contact. How you specify this endpoint is up to you, I have not made any restrictions there. So it can be source="obs:..." or source="https://..." or source="registry.opensuse.org" or source="local:..." ... point is if the source is understood depends on the given backend name. This gives us some flexibility to add more

[travier]

I was not quite clear what you mean by specify that you want to take it from the current local storage ? which local storage ? the one of the build host you build the image on ?

Yes, that would be the build host's docker or podman image store. If I know that I'm going to build X images with a specific container in it, instead of pulling it X times into each of the images, I would pull it once in my local storage and then point the XML config to it instead.

Or that could be for images that are not pushed to a registry at all.

[rjschwei]

A few more wording and description suggestions. LGTM

[schaefi]

Yes, that would be the build host's docker or podman image store. If I know that I'm going to build X images with a specific container in it, instead of pulling it X times into each of the images, I would pull it once in my local storage and then point the XML config to it instead.

ok this means your local system needs a registry server e.g the CNCF distribution registry configured as a proxy. You would pull your container once and in your image build you could reference via source="localhost:5000" as an example for a registry server on port 5000. That would work this way without changes on the kiwi side once this PR is merged

Or that could be for images that are not pushed to a registry at all.

right which means fetch_only="true"

So I guess this use case would be covered. There is just not much influence where in the system the container would be stored and the current selection of backends would allow for the format to be oci-archive only. But as I said we can add more granular settings later when required

[travier]

Yes, that would be the build host's docker or podman image store. If I know that I'm going to build X images with a specific container in it, instead of pulling it X times into each of the images, I would pull it once in my local storage and then point the XML config to it instead.

ok this means your local system needs a registry server e.g the CNCF distribution registry configured as a proxy. You would pull your container once and in your image build you could reference via source="localhost:5000" as an example for a registry server on port 5000. That would work this way without changes on the kiwi side once this PR is merged

Sure, that could work, but it's super inefficient. If I already have the OCI archive on the build host, copying it into the rootfs of the destination image is "free" for filesystems that do copy-on-write for example. Note that all those transports are supported by skopeo for example so it's not code in kiwi.

Or that could be for images that are not pushed to a registry at all.

right which means fetch_only="true"

I was more thinking about images that you've just built locally and are on the build host but not a in registry. It's independent of how you would import them in the final system.

[schaefi]

I was more thinking about images that you've just built locally and are on the build host but not a in registry. It's independent of how you would import them in the final system.

sure and to add this data into an image you don't need anything from what we are discussing here. I'm not sure if we are following the same line of thoughts. The feature provided in this PR is all about making container(s) from registrie(s) available in the local storage of an image such that a backend system be it podman, docker, containerd, etc etc can consume it without the need to pull them in first. If you have everything available on your local machine and just want to share data with an image build you can do this in various ways but it will also only work on this machine and nowhere else. Thus there is no reason to specify a source definition in a kiwi description, you could just go with a kiwi script that runs some cp, skopeo or podman image save or some other command to get the data into the image.

Maybe we can take this topic offline ? It confuses me to be honest and I'm not sure if it helps with the review of the PR to be merged

For me the most important part is, if we all agree that the current description schema serves the purpose for the feature for which @rjschwei has opened a request for and that it can be used/adapted to other scenarios that @Conan-Kudo and @travier have in mind ? I mean it's not all that of a rocket science, we fetch containers and load them and the description elements to describe this should be flexible enough to fetch from everywhere and to load into anything, which I believe is possible the way we describe it.

Thoughts ?

[rjschwei]

The implementation in this PR works for me and covers #2663 as far as I am concerned. We have some immediate use cases and a couple in the not too distant future, as such getting this over the finish line soon would be great. Thanks @schaefi for the effort an dthe quick implementation. @Conan-Kudo any other concerns?

[Conan-Kudo]

LGTM and I merged it. It's extensible enough that I think we can add Incus and Flatpak with minimal fuss down the road.