Inconsistent docker image mediatypes after caddy 2.7.5

[netsandbox]

I currently debug a problem with our internal Quay container registry, where I can't mirror caddy images after 2.7.5 anymore.

With caddy 2.7.5, the docker image manifests had a consistent mediaType of application/vnd.docker.distribution.manifest.v2+json:

$ docker manifest inspect caddy:2.7.5
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1158,
         "digest": "sha256:0eff0e7322c718fc65b783418531f1e75c7f27f1a3877823378737e91296ee8c",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1158,
         "digest": "sha256:e05dc14b4b6c6dbdcfbdbe28c73e86cafc5287fa13d4f51fdfdfad7b51b8936d",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v6"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1158,
         "digest": "sha256:c3be58f3d1b5614d2a5e7bff50bc07567044bb119ec0f286c94df8169efc313b",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v7"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1158,
         "digest": "sha256:e7f68eb0f76a55ada8be1bb5213e5bbbcb0727fa8d4051f6f3edf1275bc2dc40",
         "platform": {
            "architecture": "arm64",
            "os": "linux",
            "variant": "v8"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1158,
         "digest": "sha256:3f9f202dc07f3bb097e0e78330d59f52c488ff4ba3b4aa4a569f1a28c9db95f8",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1158,
         "digest": "sha256:159a339dbc4d0e8279faad422e0a871d82adcc6736b9c4ac4c4b252c3dbee6a3",
         "platform": {
            "architecture": "s390x",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 4912,
         "digest": "sha256:38df49f40c98b8d866440b7fec4635fcf669f4c138fd001b95f457b42e4e2265",
         "platform": {
            "architecture": "amd64",
            "os": "windows",
            "os.version": "10.0.17763.5122"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 4912,
         "digest": "sha256:7a86ad9fb59b43cc97f50c3516b4153bb228276c026e4c802c4a213b0fe1c20f",
         "platform": {
            "architecture": "amd64",
            "os": "windows",
            "os.version": "10.0.20348.2113"
         }
      }
   ]
}

This changed with 2.7.6, where all manifests has a mediaType of application/vnd.oci.image.manifest.v1+json,
except for os windows, where the mediaType is application/vnd.docker.distribution.manifest.v2+json:

$ docker manifest inspect caddy:2.7.6
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.oci.image.index.v1+json",
   "manifests": [
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 1914,
         "digest": "sha256:7b51768d110708c44179dc299884e9ee73d243a37abccce2dc796abc36371a38",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 840,
         "digest": "sha256:666c65a5f3beaff567eaca64d4f307c64b81695485a64ad5d33ac192c1b299e4",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 1916,
         "digest": "sha256:a37e44b94565539d082d85ceab947792effbecc9d7db223ea6abca196ee05a90",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v6"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 567,
         "digest": "sha256:29db70769c20c963daa30568350c31fc72970c036d8e8a0d7a7a8a47d275300e",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 1916,
         "digest": "sha256:3d1e889eda417007d88637f9adca7509e5bb547492e555d4a758ce2df850a33f",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v7"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 840,
         "digest": "sha256:1f74bd28526188e78a3ee7ac25ba3de34683b3251d1e0b495cc33270910d6d64",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 1916,
         "digest": "sha256:578dcc962eb6776e3e6ed964d11a13d40983f294e590f0040fc9febce144fa3f",
         "platform": {
            "architecture": "arm64",
            "os": "linux",
            "variant": "v8"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 840,
         "digest": "sha256:38001a08c78526af0f51525a1f82a25923b94559287782c232932bff2c9a6682",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 1916,
         "digest": "sha256:2c0050449fe45c8df3fa0b90fa1621dd8fb2de864d8154aa379c73b71d1ee769",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 840,
         "digest": "sha256:09f14bafcec41acaf23fa3fbfcb3542d754279ff9bcd6994883dd551a91d9118",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 1914,
         "digest": "sha256:faab60dc3a1acc475e2d0f12c86d9f31290b49896611d5cd61feb631ea7b0b32",
         "platform": {
            "architecture": "s390x",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 840,
         "digest": "sha256:a941d2a34e67bad02c9ee6d9cb55c09fccf5ae73670d0c5cf653ba2457ae8cd7",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 4912,
         "digest": "sha256:e2f8b997d964717694810a581f3548bb7c88de368a40ad9de4f8ed414fa2c8bc",
         "platform": {
            "architecture": "amd64",
            "os": "windows",
            "os.version": "10.0.20348.2461"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 4912,
         "digest": "sha256:39ca8d38bb88e8ba8d04cd6eeeff4530e4db12daa30aa37e9a0c216a3e2bcfcd",
         "platform": {
            "architecture": "amd64",
            "os": "windows",
            "os.version": "10.0.17763.5820"
         }
      }
   ]
}

I actually don't know the container image specification, if it is allowed to have application/vnd.oci.image.manifest.v1+json and application/vnd.docker.distribution.manifest.v2+json manifest in the same image, or if our Quay container registry should accept such images.

golang, docker and traefik for example build their Linux and Windows docker images as separate tags. Maybe for a reason?

[francislavoie]

This isn't managed in this repo, those tags etc are managed by the Docker Official Images build.

/cc @tianon do you know what's going on here? I don't know about Docker image mediaType

[tianon]

This is a known but somewhat unfortunate side effect.

We've been slowly moving images to OCI media types (more accurately, building with BuildKit, which moves them to the OCI media types), but some images unfortunately can't be built with BuildKit (Windows being notable). See docker-library/python#953 (comment) for a similar discussion (opencontainers/image-spec#1150 and quay/quay#1959 are both very relevant).

FWIW, 2/3 of your examples just happen to be exceptions; most other DOI also have this problem for tags which contain both Windows and Linux: 😅

• https://oci.dag.dev/?image=eclipse-temurin@sha256:5ad4efff3364b06c61578b267138359bcba92acc20dfd533f35b75c709a6f10b
  (eclipse-temurin:latest)
• https://oci.dag.dev/?image=golang@sha256:ad5c126b5cf501a8caef751a243bb717ec204ab1aa56dc41dc11be089fafcb4f
  (golang:latest)
• https://oci.dag.dev/?image=haxe@sha256:7b7341228f9c545ac616a4e6715f7eb5282d326dc3e83c2b01de66a6ba0eb8e3
  (haxe:latest)
• https://oci.dag.dev/?image=hello-world@sha256:d211f485f2dd1dee407a80973c8f129f00d54604d2c90732e8e320e5038a0348
  (hello-world:latest)
• https://oci.dag.dev/?image=hylang@sha256:7ed8658f78db5e537e2aa13911b9007f9d530454d3e83c7148d91c30e6d6addf
  (hylang:latest)
• https://oci.dag.dev/?image=julia@sha256:59ccf7b59abf7f7d78813b5fb0e244b0c7830c5656b956f9fde7bd27626476c0
  (julia:latest)
• https://oci.dag.dev/?image=mongo@sha256:3984cf5a234e525253619060fcbff12449db0597d62a6d4e18991a18f2365c36
  (mongo:latest)
• https://oci.dag.dev/?image=pypy@sha256:a03c672c1cafb39d6a1e86989ddf027c5de8da351a1c43cf7045e4c1af335ce1
  (pypy:latest)
• https://oci.dag.dev/?image=python@sha256:a31cbb4db18c6f09e3300fa85b77f6d56702501fcb9bdb8792ec702a39ba6200
  (python:latest)

[francislavoie]

Thanks for the context! I'll close this, nothing actionable for us here.