docker-registry has changed its JWT implementation, no longer supports libtrust key IDs

[zfLQ2qx2]

It looks like docker-registry made a breaking change on Oct 19th, 2023:

distribution/distribution#4096

As a result the kid field of the JWT token is no longer valid - the code to process the the values produced by docker-libtrust has been removed from docker-registry completely. It is not clear if the kid format just needs to be updated or if there are additional fields needed.

It looks like the last docker-registry commit that will work with the current docker-auth is 1d410148efe6d1b7fd56457507a9dd465b105ec4

[mikecook]

Note that the breaking change is in v3 but not in v2 of registry.
see https://github.com/distribution/distribution/releases

[eduardgruy]

I figured out how to work around this. Basically you need to use the JWKS parameter from https://distribution.github.io/distribution/about/configuration/#token

It takes a while but bear with me:
First, you are here because you get an error like:
msg="failed to verify token: token signed by untrusted key with ID: "WMCQ:S6WN....:L26Y:TIIM""

Note down the keyID because you will need it later.

Now how to create the JWKS file:

1. Generate your certificate and key
2. Use this script to generate part jwks.json

# rfc7638-rsa-thumbprint.bash

# Requirements:
# - perl >= 5.26.1
# - base64
# - sha256sum
# - openssl

INPUT_CERT=${1??'Missing certificate'}

# Helping hands from perl
# - Remove any newlines
# - Even the hex text length if length is odd
# - Split in groups of two chars
# - Get the integer value of the two hex chars (FF => 255)
# - Convert the integer value to a real byte
# - Print it all without separators
function hex_to_bytes {
    perl -00 -nwE 'chomp($_); $_ = "0".$_ if (length($_) % 2) == 1; print map { pack("C", $_) } map { hex($_) } ($_ =~ /(..)/g);'
}

# base64url means:
#   plus  => minus
#   slash => underscore
#   no padding (=)
function b64url {
    base64 -w0 | tr '+' '-' | tr '/' '_' | tr -d '='
}

# Find where the public key parameters section is
PUBKEY_OFFSET=$(openssl asn1parse -in "$INPUT_CERT" -strictpem -inform DER -i | sed -ne '/:rsaEncryption/,/BIT STRING/p' | tail -n1 | cut -f1 -d: | xargs)
PUBKEY_PARAMS=$(openssl asn1parse -in "$INPUT_CERT" -strictpem -inform DER -strparse $PUBKEY_OFFSET -item RSAPublicKey)

# Extract modulus and exponent
MODULUS=$(echo "$PUBKEY_PARAMS" | grep -Po "(?<=n: ).+"| hex_to_bytes | b64url)
EXPONENT=$(echo -n "$PUBKEY_PARAMS" | grep -Po "(?<=e: ).+"| hex_to_bytes | b64url)

# Build the Thumbprint for the KID
# https://datatracker.ietf.org/doc/html/rfc7638#section-3.1
#
# Parameter definitions from
# https://datatracker.ietf.org/doc/html/rfc7518#section-6.3.1
KID=$(printf '{"e":"%s","kty":"RSA","n":"%s"}' $EXPONENT $MODULUS | sha256sum | perl -ae 'print $F[0]' | hex_to_bytes | b64url)

# Print the JWK to consume with distribution
printf '{
    "e": "%s",
    "kid": "%s",
    "kty":"RSA",
    "n":"%s"
}' $EXPONENT $KID $MODULUS

You will get output like:

bash jwks.sh certificate.pem
{
    "e": "AQAB",
    "kid": "8BWI....VHRiI",
    "kty":"RSA",
    "n":"jttfG........drilrcE"
}

use the output to form a json file jwks.json like:

{
"keys":[        {
    "e": "AQAB",
    "kid": "8BWI....VHRiI", <---- Replace this with your KeyID from error message
    "kty":"RSA",
    "n":"jttfG........drilrcE"
}]
}

Example setup:

version: '3.8'

services:
  registry-auth:
    image: cesanta/docker_auth:1
    volumes:
      - ./auth_config.yml:/config/auth_config.yml      <----- just use your certificate and key as you would normally
      - ./certs:/certs
  registry:
    image: registry:3.0.0-rc.1
    environment:
      - REGISTRY_AUTH=token
      - REGISTRY_AUTH_TOKEN_REALM=https://registry-auth:5000/auth
      - REGISTRY_AUTH_TOKEN_SERVICE="Docker registry"
      - REGISTRY_AUTH_TOKEN_ISSUER="Acme auth server"
      - REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE=/certs/certificate.pem
      - REGISTRY_AUTH_TOKEN_JWKS=/certs/jwks.json   <------ JWKS param
      - REGISTRY_HTTP_ADDR=0.0.0.0:5000
    ports:
      - "5000:5000"
    volumes:
      - ./certs/certificate.pem:/certs/certificate.pem
      - ./certs/jwks.json:/certs/jwks.json   <------ Inject the jwks file
      - auth_data:/auth
      - certs_data:/etc/certs
    depends_on:
      - registry-auth

volumes:
  auth_data:
  certs_data:

Based on this issue
distribution/distribution#4470

[zfLQ2qx2]

@eduardgruy I can't wait to test this! I ended up writing a replacement for docker_auth just so I could start to transition to registry v3.x but I didn't get very far because with the RFC style kid it stops complaining about the key id, but just gives a permission denied message.

[brandond]

I used the script above to generate a jwks.json file but it does not seem to have helped. There is additional discussion ongoing at distribution/distribution#4487

[brandond]

I have no idea how the above worked for anyone, it is not the correct format for the jose.JSONWebKeySet struct that is expected to be loaded from the file: https://github.com/go-jose/go-jose/blob/fdc2ceb0bbe2a29c582edfe07ea914c8dacd7e1b/jwk.go#L331-L334

It should contain a top-level keys field that is a list of keys:

{
  "keys": [
    {
      "e": "AQAB",
      "kid": "KX3Z:...:C5FZ",
      "kty": "RSA",
      "n": "xZ...CWQ"
    }
  ]
}

EDIT: I see that this is indeed documented above, but for some reason the script doesn't output it in that format? Maybe my haste will help someone else.

[brandond]

On the apparently likely chance that the distribution maintainers are really going to proceed with releasing distribution/v3 with breaking changes for auth provider support, I have tossed together a quick project to generate a JWKS file from a CA bundle. Any auth providers that have not yet migrated to generating RFC7638 key IDs, and are still using libtrust keyIDs, will need to provide both the CA bundle and the JWKS file.

https://github.com/brandond/bundle2jwks