Add Proposal F

[jdolitsky]

Considering unfinished discussion in #47 and #48, and also on the weekly call yesterday, the best path forward is to add another proposal (F) that addresses these concerns. Please see the template.

Without an alternative proposal to consider, the WG will continue to explore the feasibility of Proposal E, our "frankenstein / voltron" proposal that, to the best of our abilities, combines various good parts of proposals A-D.

Please let us know if we can expect to see this soon.

cc: Docker friends @chris-crone @tonistiigi @errordeveloper @justincormack

[chris-crone]

👋 Before following the full blown template, I thought it'd be quicker to give an outline of a possible solution. It has elements of Proposal C and D, and tackles signing and attestations.

Goals:

• Do not require new object types or fields
• Do not require new registry APIs
• Reasonable performance for verification so that security can be enforced by default

Non-goals:

• Define exact mediatypes for attestations

The unsigned object that you want to attach metadata to is represented by an index or image manifest. Metadata is stored in a signed index that references the unsigned index in the "manifests" section [see footnote 0 about nested indexes]. s/Docker types/OCI types/g for this example.

graph TD;
    SignedIndex-->UnsignedIndexOrManifest;

Loading

Unsigned index or manifest (sha256:6444e2ab69f4c3acedad9997be5e3bdc498edec1dc7b3a5dac8faed54899ef16) - a truncated Alpine image:

{
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "schemaVersion": 2,
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "digest": "sha256:c3c58223e2af75154c4a7852d6924b4cc51a00c821553bbd9b3319481131b2e0",
         "size": 528,
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "digest": "sha256:4ff3ca91275773af45cb4b0834e12b7eb47d1c18f770a0b151381cd227f4c253",
         "size": 528,
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      }
   ]
}

Signed index pushed with unsigned object's tag (ccrone/test:oci) or, optionally if you want unsigned index digest -> metadata lookup <repo>:<alg>-<unsigned_ref>.signed (ccrone/test:sha256-6444e2ab69f4c3acedad9997be5e3bdc498edec1dc7b3a5dac8faed54899ef16.signed):

{
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "schemaVersion": 2,
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json", 
         "digest": "sha256:6444e2ab69f4c3acedad9997be5e3bdc498edec1dc7b3a5dac8faed54899ef16", // Unsigned index (Alpine)
         "size": 741,
         "annotations": {
             "dev.cosignproject.cosign/signature.0": "...",
             "dev.sigstore.cosign/bundle.0": "...",
             "dev.sigstore.cosign/certificate.0": "...",
             "dev.sigstore.cosign/chain.0": "...",
             "dev.cosignproject.cosign/signature.1": "...",
             "dev.sigstore.cosign/bundle.1": "...",
             "dev.sigstore.cosign/certificate.1": "...",
             "dev.sigstore.cosign/chain.1": "..."
        }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "digest": "sha256:5b0044a1244...", // arm64 SBOM
         "size": 1024,
         "annotations": {
             "org.opencontainers.reference.type": "sbom",
             "org.opencontainers.reference": "sha256:c3c58223e2af75154c4a7852d6924b4cc51a00c821553bbd9b3319481131b2e0"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "digest": "sha256:5b0044a44d...", // amd64 SBOM
         "size": 1024,
         "annotations": {
             "org.opencontainers.reference.type": "sbom",
             "org.opencontainers.reference": "sha256:4ff3ca91275773af45cb4b0834e12b7eb47d1c18f770a0b151381cd227f4c253"
         }
      }
   ]
}

Note that the signature data is stored as indexed annotations. Signing tools would need to increment n and append their signature to the list. Storing them on the root object runs into the editing indexes is racy issue but allows for tracking signing history and much quicker verification. I think that the OCI should solve the racy index edit issue as it's causing pain here and elsewhere.

[footnote 0] This assumes client support for nested indexes. If this is not a safe assumption, the contents of the subject's "manifests" can be copied into the signed index. Verification tools should then check the signed and unsigned indexes contain the same objects.

[jdolitsky]

Note that the signature data is stored as indexed annotations

Wondering if this part will cause some controversy, but looking forward to the full proposal! 🍿🍿🍿

[tonistiigi]

@jdolitsky Is there some previous discussion about this that we can read? It is mostly for performance to keep the verification overhead minimal compared to unsigned images. If you are just worried about some fields (eg. identity.docker-reference) being unsigned now then payload can still be under annotations.

[dlorenc]

Note that the signature data is stored as indexed annotations. Signing tools would need to increment n and append their signature to the list. Storing them on the root object runs into the editing indexes is racy issue but allows for tracking signing history and much quicker verification. I think that the OCI should solve the racy index edit issue as it's causing pain here and elsewhere.

Edit: I misinterpreted what root object meant. Original text preserved below.

I agree we should solve racy edits. I think the incrementing approach can work fine, and is a reasonable tradeoff given the requirement on not adding new fields. A list would obviously be "cleaner", but I'm not a purist. If we go this route we should specify how to handle gaps in the integers and move on.

---BEGIN ORIGINAL TEXT---
We evaluated both of these in cosign and settled on accepting the race conditions in the hope that the OCI could solve those. I can't remember all the details but the incrementing approach was definitely a very close second.

[dlorenc]

Overall I like this but need to dig more into compatibility today. The early attempts at cosign were very similar but I think we ran into some compatibly issues with registries.

Some cases we should test:

• Creating an OCI index that points to images with Docker media types (I think some registries didn't like this, @jonjohnsonjr might remember more).
• Actually that's all I can remember.

[imjasonh]

This looks really promising!

Note that the signature data is stored as indexed annotations. Signing tools would need to increment n and append their signature to the list. Storing them on the root object runs into the opencontainers/distribution-spec#250 but allows for tracking signing history and much quicker verification. I think that the OCI should solve the racy index edit issue as it's causing pain here and elsewhere.

I don't love it -- I don't think any of us do -- but if this is the biggest wart then I'll take it. 😆

+1 to making OCI specify handling racy updates better, that's needed regardless.

If we specify this we should take care when wording the incrementing n behavior. What if the manifest has .1, .2, .4 (e.g., a previous signature was removed, maybe incorrectly). Should the next one be .5 ("take the highest N and add 1") or .3 ("count until you find a gap then fill it"). I don't think it matters either way, but it affects how verifiers behave -- should they iterate until they don't find an N+1, or find all .Ns regardless of gaps? Probably the latter, we just need to be specific.

---

Re:

      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "digest": "sha256:5b0044a1244...", // arm64 SBOM
         "size": 1024,
         "annotations": {
             "org.opencontainers.reference.type": "sbom",
             "org.opencontainers.reference": "sha256:c3c58223e2af75154c4a7852d6924b4cc51a00c821553bbd9b3319481131b2e0"
         }
      },

Some questions:

• Should we catalog common types? I don't think we want to make these be "official" media types, but we should probably specify what "type: sbom" means.
• In this example, is the reference referring to a blob containing the SBOM data? Maybe reference.blob would be clearer.
• What is the descriptor digest referring to? Another image manifest with a pointer to the reference blob? If both are specified inline just to avoid a lookup, I totally get that, but it means there's a manifest that exists only to be ignored. 🤷‍♂️

---

[footnote 0] This assumes client support for nested indexes. If this is not a safe assumption, the contents of the subject's "manifests" can be copied into the signed index. Verification tools should then check the signed and unsigned indexes contain the same objects.

When you say they "can" be, do you mean you expect they practically will be? I think we should take a stronger stand on which option we want to encourage or require, instead of specifying that both are possible, since it means clients have to check and handle both behaviors.

edit: Rereading this section, I misread this as "clients can do either" and not "if support for nested indexes can't be safely assumed, we can update the proposal to copy the unsigned index's manifests into the signed index." This means we'd change the digest whenever we add a new attachment, which has different understood downsides. Let me know if I'm still misreading or misinterpreting.

[chris-crone]

Thanks for the reviews so far!

@dlorenc

I think the incrementing approach can work fine, and is a reasonable tradeoff given the requirement on not adding new fields. A list would obviously be "cleaner", but I'm not a purist. If we go this route we should specify how to handle gaps in the integers and move on.

100 % agree. If there are better ideas for how to handle this, I'd be happy to hear them.

Creating an OCI index that points to images with Docker media types (I think some registries didn't like this, @jonjohnsonjr might remember more).

We had the same issue with storing CNABs. We might even have a mapping of which registry supported what somewhere. We solved it by using a fallback to Docker types. I'm the first to admit that this is not ideal but we decided it was the least worst option. Rewriting manifests in a different format is an option but it will change digests as well.

I would be curious to hear about other pitfalls that you ran into though.

@imjasonh

I don't love it -- I don't think any of us do -- but if this is the biggest wart then I'll take it. 😆

This is a great summary of how I feel 😬 It really is a case of trying to pick the least worst option. If it's any consolation, it should mostly be us tool builders who're exposed to this not end users.

Should we catalog common types? I don't think we want to make these be "official" media types, but we should probably specify what "type: sbom" means.

Yes, we'd likely want some well known types list for interoperability. I think the OCI is an appropriate place to define this.

In this example, is the reference referring to a blob containing the SBOM data? Maybe reference.blob would be clearer.

I was intentionally ambigious here. I know that the OCI have previously raised that they don't like indexes pointing to blobs from the "manifests" list and want items in the list to point to manifests. Using a manifest adds a round trip for the client but could allow interesting data structures for storing SBOMs (e.g.: ones broken up by layer?).

What is the descriptor digest referring to? Another image manifest with a pointer to the reference blob? If both are specified inline just to avoid a lookup, I totally get that, but it means there's a manifest that exists only to be ignored.

      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "digest": "sha256:5b0044a1244...", // arm64 SBOM
         "size": 1024,
         "annotations": {
             "org.opencontainers.reference.type": "sbom",
             "org.opencontainers.reference": "sha256:c3c58223e2af75154c4a7852d6924b4cc51a00c821553bbd9b3319481131b2e0"
         }
      },

In the example, .digest points to a manifest descriptor that contains the SBOM and .annotations.org.opencontainers.reference points to which image the SBOM is for. This allows us to keep the list flat and associate objects together.

edit: Rereading this section, I misread this as "clients can do either" and not "if support for nested indexes can't be safely assumed, we can update the proposal to copy the unsigned index's manifests into the signed index." This means we'd change the digest whenever we add a new attachment, which has different understood downsides. Let me know if I'm still misreading or misinterpreting.

I didn't want to distract from the main proposal which is why I added this as a footnote. Today, some runtimes ignore nested indexes and try to find a manifest that meets their needs in the top level index. This would mean with the above design, they would fail to run an image.

The workaround is to copy the manifests section from the nexted index into the signed top level index. This would mean that runtimes that don't traverse the nested index would find an image to run and work. When appending signatures, the original object (the unsigned index) would remain the same but the signed index would be mutated with new annotations. This would mean that the original object's digest would remain constant.

One possible issue with this is what should client tooling do if the manifests in the signed index do not match those in the nested index? I think this is solvable though. We just need to write down what the behavior should be.

[sudo-bmitch]

Looking over this, I've got a couple questions:

1. If someone pulls alpine@sha256:6444e2ab69f4c3acedad9997be5e3bdc498edec1dc7b3a5dac8faed54899ef16, how do they lookup the metadata pointing to that digest?

2. What prevents an existing registry that isn't aware of the annotations from performing a GC on the sbom manifest (e.g. sha256:c3c58223e2af75154c4a7852d6924b4cc51a00c821553bbd9b3319481131b2e0)?

[tonistiigi]

If someone pulls alpine@sha256:6444e2ab69f4c3acedad9997be5e3bdc498edec1dc7b3a5dac8faed54899ef16, how do they lookup the metadata pointing to that digest?

From above:

Signed index pushed with unsigned object's tag (ccrone/test:oci) or, optionally if you want unsigned index digest -> metadata lookup :-<unsigned_ref>.signed (ccrone/test:sha256-6444e2ab69f4c3acedad9997be5e3bdc498edec1dc7b3a5dac8faed54899ef16.signed):

From that index client can find the attestation or signature for any inner object.

What prevents an existing registry that isn't aware of the annotations from performing a GC on the sbom manifest (e.g. sha256:c3c58223e2af75154c4a7852d6924b4cc51a00c821553bbd9b3319481131b2e0)?

sha256:c3c58223e2af is listed in the manifest array of sha256:6444e2ab6. Note that this is not the object that contains sbom. That object is sha256:5b0044a1244.

[sudo-bmitch]

If someone pulls alpine@sha256:6444e2ab69f4c3acedad9997be5e3bdc498edec1dc7b3a5dac8faed54899ef16, how do they lookup the metadata pointing to that digest?

From above:

Signed index pushed with unsigned object's tag (ccrone/test:oci) or, optionally if you want unsigned index digest -> metadata lookup :-<unsigned_ref>.signed (ccrone/test:sha256-6444e2ab69f4c3acedad9997be5e3bdc498edec1dc7b3a5dac8faed54899ef16.signed):

From that index client can find the attestation or signature for any inner object.

Woops, completely overlooked that line. We should consider a more generic name since this will be used for more than just signing.

What prevents an existing registry that isn't aware of the annotations from performing a GC on the sbom manifest (e.g. sha256:c3c58223e2af75154c4a7852d6924b4cc51a00c821553bbd9b3319481131b2e0)?

sha256:c3c58223e2af is listed in the manifest array of sha256:6444e2ab6. Note that this is not the object that contains sbom. That object is sha256:5b0044a1244.

Got it. Thanks!

Sounds like the big concerns have already been highlighted, race conditions, and existing runtimes that won't follow nested Indexes. The latter is easy enough to work around by not changing the original tag and using digest tags, or pulling up platform specific descriptors.

The reason that D opted for unique digest tags was to avoid the race conditions, so we're left deciding if we want to not solve race conditions to simplify the client pull.

We should also test how existing runtimes deal with an Index that includes multiple descriptors without any platform specification.

[errordeveloper]

I think the incrementing approach can work fine, and is a reasonable tradeoff given the requirement on not adding new fields. A list would obviously be "cleaner", but I'm not a purist. If we go this route we should specify how to handle gaps in the integers and move on.

100 % agree. If there are better ideas for how to handle this, I'd be happy to hear them.

Just a few ideas we can explore...

Top-level key holds a JSON list

{
         "annotations": {
             "org.opencontainers.signature": "[{ \"name\": \"dev.cosignproject.cosign\", \"signature\": \"...\", \"bundle\": \"...\", \"certificate\": \"...\" },  { \"name\": \"dev.sigstore.cosign\", \"chain\": \"...\" }, { \"name\": \"dev.cosignproject.cosign\", \"signature\": \"...\" },  { \"name\": \"dev.sigstore.cosign\", \"bundle\": \"...\" }, { \"name\": \"dev.sigstore.cosign\", \"certificate\": \"...\" }, { \"name\": \"dev.sigstore.cosign\", \"chain\": \"...\" }]"
        }
}

Hashes in keys and a separate index

This replaces use of ints in @chris-crone's proposal with hashes of each of the value and adds an explicit index.
I've used CSV for the index format here, to keep it simple, but of course it can be a JSON list.

{
         "annotations": {
             "dev.cosignproject.cosign/signature.e59879c": "...",
             "dev.sigstore.cosign/bundle.6436749": "...",
             "dev.sigstore.cosign/certificate.776a3f1": "...",
             "dev.sigstore.cosign/chain.8adf60e": "...",
             "dev.cosignproject.cosign/signature.de25bf6": "...",
             "dev.sigstore.cosign/bundle.37d34ff": "...",
             "dev.sigstore.cosign/certificate.e8adf60": "...",
             "dev.sigstore.cosign/chain.b1c64a3": "...",
             "org.opencontainers.signatures.index":"dev.cosignproject.cosign/signature.e59879c,dev.cosignproject.cosign/signature.de25bf6,dev.sigstore.cosign/bundle.37d34ff,dev.sigstore.cosign/certificate.e8adf60,dev.sigstore.cosign/chain.b1c64a3,dev.sigstore.cosign/bundle.6436749,dev.sigstore.cosign/certificate.776a3f1,dev.sigstore.cosign/chain.8adf60e",
        }
}

---

I mostly just thought to put a few ideas foward, so we have something to discuss. I do think it'd best if implementations didn't have to parse to worry about which one is the number to pick and increment.

[justincormack]

Yes, integers has a lot of problems, just using an identifier is much better. Could use the public key hash.

[dlorenc]

We had the same issue with storing CNABs. We might even have a mapping of which registry supported what somewhere. We solved it by using a fallback to Docker types. I'm the first to admit that this is not ideal but we decided it was the least worst option. Rewriting manifests in a different format is an option but it will change digests as well.

Fallback would work fine, but don't we then have an issue with the Docker manifest list not having an annotation field? https://docs.docker.com/registry/spec/manifest-v2-2/#example-manifest-list

[nishakm]

Note that the signature data is stored as indexed annotations

Wondering if this part will cause some controversy, but looking forward to the full proposal! popcornpopcornpopcorn

My concern with embedding this kind of data in annotations is that OCI annotations are optional and do not follow any set format. So if a client doesn't understand or ignores these annotations, all the reference data will be lost, which is no different from the backwards compatibility issues that were raised in Proposal C (moving manifests from V3 to V2 registries will result in loss of references).

[nishakm]

Reg: racey index: currently, one can GET and PUT any manifest by digest using /v2/<name>/manifests/<reference>. Doesn't this mean an updated index manifest is a new digest and hence not racey?