Preserve manifest list digest when embedding containers

[achilleas-k]

Skopeo source

This changes the intermediate format of a container source from docker-archive to dir, a directory format that's not yet standardised, but compatibility guidelines are currently being discussed (containers/image#1874).

The dir format lets us merge in a multi-image manifest with a container so that we can preserve the manifest digest that was used to specify the image. This lets users run the image using the same digest that they used to specify it for pulling. See containers/skopeo#1935 for details on the merge method.

Skopeo-index source

New source for specifying the manifest-list to download. This can then be used as a secondary (optional) input to the skopeo stage (see below). The output of the source is a single file, the manifest that the digest points to. If the digest points to any other type of object, the source fails.

Skopeo stage

The skopeo stage now takes an optional list of file references as input. These files must be manifest-lists. Each manifest-list is automatically paired with an image source based on the single-image manifest digest.

Example

Using the Fedora 37 image on Quay.io as an example, we can now specify the following sources:

    "org.osbuild.skopeo": {
      "items": {
        "sha256:38b8a6c320f32c32d41046e62496fdde0c712a6491a5327146fa7189a23399ad": {
          "image": {
            "name": "quay.io/fedora/fedora",
            "digest": "sha256:b00bf6a9f41f2245ad0c7fccc3a948dbb6266ae808b3f9a8977e9634b3aabdcc"
          }
        }
      }
    },
    "org.osbuild.skopeo-index": {
      "items": {
        "sha256:d61102cd2dfcb5ac29f752554c8738631245acefb02cae741562349a471fc4d3": {
          "image": {
            "name": "quay.io/fedora/fedora"
          }
        }
      }
    }

• sha256:d61102cd2dfcb5ac29f752554c8738631245acefb02cae741562349a471fc4d3: Manifest-list digest. This is the multi-image manifest that (in this case) refers to four single-image manifests, one for each architecture. This is used to download the manifest-list as a single file by the skopeo-index stage.
• sha256:b00bf6a9f41f2245ad0c7fccc3a948dbb6266ae808b3f9a8977e9634b3aabdcc: Image manifest digest. This is the single-image manifest for the amd64 architecture. This is used to download the image by the skopeo stage.
• sha256:38b8a6c320f32c32d41046e62496fdde0c712a6491a5327146fa7189a23399ad: Image config digest (the image ID). This is the ID that appears in podman image ls for the container. This is the ID that's used to store the container in the local osbuild cache and the object ref for the container in the manifest.

The skopeo stage will then look like this:

{
  "type": "org.osbuild.skopeo",
  "inputs": {
    "images": {
      "type": "org.osbuild.containers",
      "origin": "org.osbuild.source",
      "references": {
        "sha256:38b8a6c320f32c32d41046e62496fdde0c712a6491a5327146fa7189a23399ad": {
          "name": "quay.io/fedora/fedora:with-digest"
        }
      }
    },
    "manifest-lists": {
      "type": "org.osbuild.files",
      "origin": "org.osbuild.source",
      "references": [
        "sha256:d61102cd2dfcb5ac29f752554c8738631245acefb02cae741562349a471fc4d3"
      ]
    }
  },
  "options": {
    ...
  }
},

The manifest list is parsed for single-image manifest digests and the container is inspected to get its digest. If no matches are found between the two, the stage raises an exception. manifest-lists are not mandatory, but if they are specified, they should match a container in the images.

Testing

I've tested the above example manually and also tested having multiple containers with matching manifest-lists.
I intend to add tests for the expected behaviour in osbuild-composer when we add support there.

---

Old summary. No longer relevant but kept for posterity.

This PR adds support for writing image names to the names array of the container store for images that are imported through skopeo.

Problem

• Users want to add a container to the blueprint as <url>@<digest> and then run the container after build with podman run <url>@<digest>
• The way we embed containers in images is by first pulling the container into a docker archive skopeo copy <url>@<digest> docker-archive:<container>.tar during the source download phase of the build and then we put it in the container store skopeo copy docker-archive:<container>.tar <containerstore>:<url>
• Launching the container with podman run <url>@<digest> after booting the OS doesn't work in many cases because the <digest> is likely to change. This is especially the case when relying on the manifest-list digest (which Quay and the Red Hat catalogue present to users). This is a digest of a manifest that lists multiple container images (usually the same container for different architectures). When pulling a container using this digest, only the container for the host architecture is pulled. The local manifest is then completely different from the one used to query the registry.

Observations

• When I do a podman pull <url>@<digest>, I see that podman puts the digest in the overlay-images/images.json as an "alias" for this container image.
• Something similar happens when I copy a container directly into the store with skopeo copy <url>@<digest> <containerstore>:<url>.

Solution

• Modify our skopeo stage (which does skopeo copy) to amend the image store images.json and add the digest that was used in the blueprint as a name for the container (this PR).
• We can expose this to the user as a store_source_name (or something similar) option in the container part of the blueprint (later PR in osbuild-composer).

Questions/concerns

• It's important to note that this digest is not the hash of anything that's on disk. Users want to use this digest because the registries present it as something that uniquely identifies an image, but it should be made clear that the metadata that makes up this digest only exists on the server where this container came from.
  • From what I understand, neither skopeo nor podman actually store any of this metadata. The name they put in the images.json is just that, a name, a string that was used at some point to refer to an image that eventually resolved to the one in the store.
• I've read through the container library sources to make sure that modifying the images.json file directly is safe and doesn't need any similar modifications in other places. I tested this a few times and it seems to simply work, but I'd like a sanity check from some of the container developers at the same time to make sure.
• I've seen a few places where this digest can be stored, depending on how the container is pulled. If the container is already in the store (under a different name, maybe a tag) and then it's pulled again with the digest, the digest ends up in the names array (and the names-history at the same time). If downloaded with skopeo copy directly into the store, it's stored as the digest of the image. I chose to put it in names, that feels less direct, but I'm open to suggestions.
  • I'm not sure if the names should also be included in the names-history array at the same time.

I've asked @runcom for some guidance here. I've looked at containers/image@e1ee980 and the linked issues, but that doesn't work because we rely on docker-archive for the local intermediate storage. That's one thing we can revisit in the future, whether a different local intermediate storage might make things easier, without having any negative side effects.

[runcom]

cc @mtrmac @mheon - could give us some more guidance/check on what we're doing here? 👼

[mtrmac]

TL:DR; This approach is unworkable and must be rethought.

---

• The way we embed containers in images is by first pulling the container into a docker archive skopeo copy <url>@<digest> docker-archive:<container>.tar during the source download phase of the build and then we put it in the container store skopeo copy docker-archive:<container>.tar <containerstore>:<url>

If at all possible, don’t do that. docker-archive:, besides being slow, always loses information. Use dir: if you absolutely need a non-registry file store, and want to preserve the image as it is on in-registry.

• Modify our skopeo stage (which does skopeo copy) to amend the image store images.json

Definitely don’t do that in production. The format of images.json is an internal implementation detail and may change at any time (in fact, this month we have seriously discussed what it would take to move to SQLite).

The locking of images.json is even more of an internal implementation detail, and critical to reliability. This code does not do any locking, and may therefore interfere with any running Podman/Skopeo/… processes in unanticipated ways.

The right way to add a name to an image is podman tag, or, I suppose, skopeo copy containers-storage:$old containers-storage:$new.

and add the digest that was used in the blueprint as a name for the container (this PR).

That won’t work anyway, because readers of those images will need to actually find the manifest data that matches that digest. Just adding a name does not record the manifest data.

So, the entire premise of using docker-archive should be revisited. That’s the easiest way to get this done. (An alternative would be a custom-written tool to download the right manifests, and a custom-written Go tool to add the manifests to the store. Let’s not.)

• It's important to note that this digest is not the hash of anything that's on disk.
  …
• From what I understand, neither skopeo nor podman actually store any of this metadata.

This is incorrect. AFAICS all digest reference accesses like that will fail.

• I've read through the container library sources to make sure that modifying the images.json file directly is safe

It isn’t safe. It’s not up to me but speaking for myself, I would absolutely refuse to support any modifications not implemented by calling a (reasonably current) version of containers/storage.

---

The implementation is also not ensuring that each name is only assigned to only one image.

[achilleas-k]

TL:DR; This approach is unworkable and must be rethought.

• The way we embed containers in images is by first pulling the container into a docker archive skopeo copy <url>@<digest> docker-archive:<container>.tar during the source download phase of the build and then we put it in the container store skopeo copy docker-archive:<container>.tar <containerstore>:<url>

If at all possible, don’t do that. docker-archive:, besides being slow, always loses information. Use dir: if you absolutely need a non-registry file store, and want to preserve the image as it is on in-registry.

Thanks. I think this is probably the best way forward and probably wont require too much reworking.

• Modify our skopeo stage (which does skopeo copy) to amend the image store images.json

Definitely don’t do that in production. The format of images.json is an internal implementation detail and may change at any time (in fact, this month we have seriously discussed what it would take to move to SQLite).

I see. That's great to know. It wouldn't be a problem to rely on a shell utility that can properly write to the store, it would have to be something that's packaged for RHEL.

The locking of images.json is even more of an internal implementation detail, and critical to reliability. This code does not do any locking, and may therefore interfere with any running Podman/Skopeo/… processes in unanticipated ways.

This is not an issue in our case (the stages are run in a clean bubblewrap container and no other processes should be running), but I understand the concern. Thank you.

The right way to add a name to an image is podman tag, or, I suppose, skopeo copy containers-storage:$old containers-storage:$new.

We need an intermediate local storage because the download stage happens separately from the build stage. That said, if a dir: or an intermediate container-store: gets us around this issue, we can work with that.

and add the digest that was used in the blueprint as a name for the container (this PR).

That won’t work anyway, because readers of those images will need to actually find the manifest data that matches that digest. Just adding a name does not record the manifest data.

From some testing I did, I found that this does work (adding the name) and no manifest data was pulled (e.g., when doing podman run <name>@<manifest digest>), but perhaps this is an unintended consequence or side effect?
I couldn't find any information on where or how the manifest or manifest-list is stored. Perhaps it's not important if we do this properly (with dir or container-store), but it'd be good to know.

So, the entire premise of using docker-archive should be revisited. That’s the easiest way to get this done. (An alternative would be a custom-written tool to download the right manifests, and a custom-written Go tool to add the manifests to the store. Let’s not.)

• It's important to note that this digest is not the hash of anything that's on disk.
  …
• From what I understand, neither skopeo nor podman actually store any of this metadata.

This is incorrect. AFAICS all digest reference accesses like that will fail.

• I've read through the container library sources to make sure that modifying the images.json file directly is safe

It isn’t safe. It’s not up to me but speaking for myself, I would absolutely refuse to support any modifications not implemented by calling a (reasonably current) version of containers/storage.

Good to know.

The implementation is also not ensuring that each name is only assigned to only one image.

You're right, yes, this is an oversight.

[achilleas-k]

Adding some info after speaking with @mtrmac:

That won’t work anyway, because readers of those images will need to actually find the manifest data that matches that digest. Just adding a name does not record the manifest data.

From some testing I did, I found that this does work (adding the name) and no manifest data was pulled (e.g., when doing podman run <name>@<manifest digest>), but perhaps this is an unintended consequence or side effect? I couldn't find any information on where or how the manifest or manifest-list is stored. Perhaps it's not important if we do this properly (with dir or container-store), but it'd be good to know.

They're in the image's directory and their digests are in the big-data-digests array (example below).

# jq '.[]."big-data-digests"' storage/overlay-images/images.json
{
  "manifest": "sha256:4db63516a5990cc27f1655366bf5ea2fbde97ea6295b028f916bfc6de7d96d4c",
  "manifest-sha256:4db63516a5990cc27f1655366bf5ea2fbde97ea6295b028f916bfc6de7d96d4c": "sha256:4db63516a5990cc27f1655366bf5ea2fbde97ea6295b028f916bfc6de7d96d4c",
  "sha256:7acd9e55e8752c8a64e8569ea3e2736ceb89bb4d099bae597d202ef882e103e6": "sha256:7acd9e55e8752c8a64e8569ea3e2736ceb89bb4d099bae597d202ef882e103e6",
  "signature-4db63516a5990cc27f1655366bf5ea2fbde97ea6295b028f916bfc6de7d96d4c": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
}
# sha256sum storage/overlay-images/7acd9e55e8752c8a64e8569ea3e2736ceb89bb4d099bae597d202ef882e103e6/*
4db63516a5990cc27f1655366bf5ea2fbde97ea6295b028f916bfc6de7d96d4c  storage/overlay-images/7acd9e55e8752c8a64e8569ea3e2736ceb89bb4d099bae597d202ef882e103e6/=bWFuaWZlc3Qtc2hhMjU2OjRkYjYzNTE2YTU5OTBjYzI3ZjE2NTUzNjZiZjVlYTJmYmRlOTdlYTYyOTViMDI4ZjkxNmJmYzZkZTdkOTZkNGM=
7acd9e55e8752c8a64e8569ea3e2736ceb89bb4d099bae597d202ef882e103e6  storage/overlay-images/7acd9e55e8752c8a64e8569ea3e2736ceb89bb4d099bae597d202ef882e103e6/=c2hhMjU2OjdhY2Q5ZTU1ZTg3NTJjOGE2NGU4NTY5ZWEzZTI3MzZjZWI4OWJiNGQwOTliYWU1OTdkMjAyZWY4ODJlMTAzZTY=
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  storage/overlay-images/7acd9e55e8752c8a64e8569ea3e2736ceb89bb4d099bae597d202ef882e103e6/=c2lnbmF0dXJlLTRkYjYzNTE2YTU5OTBjYzI3ZjE2NTUzNjZiZjVlYTJmYmRlOTdlYTYyOTViMDI4ZjkxNmJmYzZkZTdkOTZkNGM=
4db63516a5990cc27f1655366bf5ea2fbde97ea6295b028f916bfc6de7d96d4c  storage/overlay-images/7acd9e55e8752c8a64e8569ea3e2736ceb89bb4d099bae597d202ef882e103e6/manifest

It shouldn't work without the data being there when the name contains a digest but it does. This is a bug.

[mtrmac]

The right way to add a name to an image is podman tag, or, I suppose, skopeo copy containers-storage:$old containers-storage:$new.

We need an intermediate local storage because the download stage happens separately from the build stage.

(I was thinking this could be done in the build stage; that’s just a way to add a tag without writing an external utility, e.g. with the skopeo copy, $old and $new would use the same store but just different image names. But that should work for names with tags, it gets complicated with digests.)

[mtrmac]

It shouldn't work without the data being there when the name contains a digest but it does. This is a bug.

containers/podman#17682

[achilleas-k]

Everything changed. I've updated the PR description based on the solutions discussed with the containers team.

[mtrmac]

(I’m unfamiliar with the code base, this is just a quick skim of the container image-related aspects.)

[thozza]

This looks great and pretty clean 👍 It is great that you figured out a reasonable solution for this 🐛 . The only thing that I'm not sure about is the ability to specify the format for a stage input. I think that it just add new options to shoot oneself in the foot for not obvious (to me) benefit.

[thozza]

I have basically the same question as for the source input origin just with the difference of asking why we should not default to the format used to produce (store) the container in a pipeline or defaulting to the "most powerful" format for storing?

[achilleas-k]

The only thing that I'm not sure about is the ability to specify the format for a stage input. I think that it just add new options to shoot oneself in the foot for not obvious (to me) benefit.

You're not wrong. It's a bit strange and creates a situation where the stage input needs to match the source. In fact, it's even weirder than that because you can specify the stage input to have a different format than the source, but if the input format was already downloaded and is in the store, it would work.

My thinking on this was that I didn't want to introduce a behaviour change without introducing an option. On the other hand, this is an internal implementation detail and it might not matter. You're right that the most powerful format right now is the dir, so we could use that everywhere, or at least only use it for sources and keep pipeline artifacts as oci-archive to minimise changes. My one issue with this that the dir format isn't standardised and I intend to use it only when this new behaviour is desired, keeping the old behaviour intact for the default case. But perhaps this is the wrong way to think about it and we could go all in with dir and bring back other formats if we ever run into incompatibilities.

I thought I was making the most cautious choice here, but you're making me reconsider now.

[achilleas-k]

Rewrote with dir replacing docker-archive and dropped the options.

[achilleas-k]

Updated the original comment again, though the Fedora digests are outdated now.

[achilleas-k]

Fixed two small issues caught by pylint.

[achilleas-k]

Speaking with @thozza we just realised: we should change the ref ID that we use in the manifest (and cache) to something else, preferably the digest, because now we can download the same container using different manifest digests. This means that we can have the same container image ID with different metadata and this should be differentiated.

[achilleas-k]

Made the changes we discussed today. Manifest digest is now stored separately in the cache and it is only merged when the input option requests it.

[achilleas-k]

Fixed a bug, added docstrings, and changed the name of the new option from manifest_digest to list_digest to make it clear that it must be the digest of a manifest list.

[thozza]

Great work (on figuring our everything and also the test case), I love it 😍

[achilleas-k]

@thozza Waiting for one more pair of eyes before merging? Maybe @supakeen?

[thozza]

@thozza Waiting for one more pair of eyes before merging? Maybe @supakeen?

Yes, I wanted to let @supakeen have a chance to review. If he is not going to do it, we can merge 😉