When pulling a signed image that is part of a image-index the image can't be pulled.

[jmarrero]

The error is seen by using rpm-ostree or bootc to do a deployment of a signed image:

ERROR Switching: Pulling: Creating importer: Failed to invoke skopeo proxy method OpenImage: remote error: A signature was required, but no signature exists

By adding:

        c.arg("--debug");

and

        println!("stdout:\n{:?}", c.stdout(Stdio::piped()).output());

before:
https://github.com/containers/containers-image-proxy-rs/blob/main/src/imageproxy.rs#L227

I was able to print
Printing the skopeo debug output which shows:

DEBU[0000] GET https://registry.redhat.io/v2/rhel9/rhel-bootc/manifests/9.4
DEBU[0000] Content-Type from manifest GET is "application/vnd.oci.image.index.v1+json"
DEBU[0000] IsRunningImageAllowed for image docker:registry.redhat.io/rhel9/rhel-bootc:9.4

...


DEBU[0000]  Using transport "docker" specific policy section registry.redhat.io
DEBU[0000] GET https://registry.redhat.io/containers/sigstore/rhel9/rhel-bootc@sha256=b73c1d3280d6e7b3a6ed9d52138e8e4dde7384dd0227deda67c97f428f97551a/signature-1
DEBU[0001] ... got status 404, as expected = end of signatures
DEBU[0001] Not looking for sigstore attachments: disabled by configuration
DEBU[0001] Requirement 0: denied, done

This does not pull any signatures.

However when we try skopeo copy we see:

DEBU[0000] GET https://registry.redhat.io/v2/rhel9/rhel-bootc/manifests/9.4
DEBU[0000] Content-Type from manifest GET is "application/vnd.oci.image.index.v1+json"
DEBU[0000] Using SQLite blob info cache at /var/lib/containers/cache/blob-info-cache-v1.sqlite
DEBU[0000] Source is a manifest list; copying (only) instance sha256:68100781edd6317a9c9f419374568a641b8402b1901740cfd38cdd20db902caf for current system
DEBU[0000] GET https://registry.redhat.io/v2/rhel9/rhel-bootc/manifests/sha256:68100781edd6317a9c9f419374568a641b8402b1901740cfd38cdd20db902caf

...

DEBU[0000]  Using transport "docker" specific policy section registry.redhat.io
DEBU[0000] GET https://registry.redhat.io/containers/sigstore/rhel9/rhel-bootc@sha256=68100781edd6317a9c9f419374568a641b8402b1901740cfd38cdd20db902caf/signature-1
DEBU[0000] GET https://registry.redhat.io/containers/sigstore/rhel9/rhel-bootc@sha256=68100781edd6317a9c9f419374568a641b8402b1901740cfd38cdd20db902caf/signature-2
DEBU[0000] GET https://registry.redhat.io/containers/sigstore/rhel9/rhel-bootc@sha256=68100781edd6317a9c9f419374568a641b8402b1901740cfd38cdd20db902caf/signature-3
DEBU[0000] GET https://registry.redhat.io/containers/sigstore/rhel9/rhel-bootc@sha256=68100781edd6317a9c9f419374568a641b8402b1901740cfd38cdd20db902caf/signature-4
DEBU[0001] GET https://registry.redhat.io/containers/sigstore/rhel9/rhel-bootc@sha256=68100781edd6317a9c9f419374568a641b8402b1901740cfd38cdd20db902caf/signature-5
DEBU[0001] GET https://registry.redhat.io/containers/sigstore/rhel9/rhel-bootc@sha256=68100781edd6317a9c9f419374568a641b8402b1901740cfd38cdd20db902caf/signature-6
DEBU[0003] GET https://registry.redhat.io/containers/sigstore/rhel9/rhel-bootc@sha256=68100781edd6317a9c9f419374568a641b8402b1901740cfd38cdd20db902caf/signature-7
DEBU[0003] GET https://registry.redhat.io/containers/sigstore/rhel9/rhel-bootc@sha256=68100781edd6317a9c9f419374568a641b8402b1901740cfd38cdd20db902caf/signature-8
DEBU[0003] GET https://registry.redhat.io/containers/sigstore/rhel9/rhel-bootc@sha256=68100781edd6317a9c9f419374568a641b8402b1901740cfd38cdd20db902caf/signature-9
DEBU[0004] ... got status 404, as expected = end of signatures
DEBU[0004] Not looking for sigstore attachments: disabled by configuration
DEBU[0004]  Requirement 0: allowed
DEBU[0004] Overall: allowed

Which correctly pulls the signatures, however this code goes thru this path:
https://github.com/containers/image/blob/8c7c58c5aacd70fe8bc25da54f966a59baf175b0/copy/copy.go#L318

This path looks at the image-index and checks signatures only for the system image.

This path as I understand we avoid, and implement our own way of pulling using the skopeo/proxy
https://github.com/containers/skopeo/blob/main/cmd/skopeo/proxy.go#L412

I am still not sure if the fix should be here or in skopeo/proxy, but my undestanding right now is that we have to implement something similar to what containers/image/copy does.

Ultimately this blocks any deployment when a more strict /etc/containers/policy.json is defined such as when using RHEL.

Currently a workaround is to use the Manifest List Digest: such as: registry.redhat.io/rhel9/rhel-bootc@sha256:68100781edd6317a9c9f419374568a641b8402b1901740cfd38cdd20db902caf

instead of the tag.

[jmarrero]

If I change the policy.json I see:

DEBU[0000] Overall: allowed                             
TRACE open_image:impl_request: completed request self=ImageProxy imgref="docker://registry.redhat.io/rhel9/rhel-bootc:9.4" self=ImageProxy method="OpenImage"
TRACE impl_request: sending request GetManifest self=ImageProxy method="GetManifest"
DEBU[0000] GET https://registry.redhat.io/v2/rhel9/rhel-bootc/manifests/sha256:68100781edd6317a9c9f419374568a641b8402b1901740cfd38cdd20db902caf 
DEBU[0001] Content-Type from manifest GET is "application/vnd.oci.image.manifest.v1+json" 
TRACE impl_request: completed request self=ImageProxy method="GetManifest"
DEBUG finish_pipe: closing pipe self=ImageProxy pipeid=8
TRACE finish_pipe:impl_request: sending request FinishPipe self=ImageProxy pipeid=8 self=ImageProxy method="FinishPipe"
TRACE finish_pipe:impl_request: completed request self=ImageProxy pipeid=8 self=ImageProxy method="FinishPipe"
TRACE impl_request: sending request GetFullConfig self=ImageProxy method="GetFullConfig"

Maybe we can change the validation to happen during or after getManifest since it looks like at that point we would have the SHA that has the signature. We fail right now on OpenImage.

Also another way to avoid this is to sign the Digest. To get the digest you must sign you can run:

skopeo inspect docker://registry.redhat.io/rhel9/rhel-bootc:9.4 | jq .Digest