Add a Proof Type for Merkle Proof Selective Disclosure to JPA

[OR13]

I took a stab at a “JWP” encoding of merkle disclosure proofs in a token format….

https://or13.github.io/jwp-mdt/#name-root-proof

The main thing I noticed was some awkwardness regarding nesting JWS inside the protected header.

it feels like JWP “Protected Header” should reserve a member “signature” and its value should not contain concatenated “header” or “payload” attributes…

 "protected": {

    "kid": "did:example:123#key-0"

    "root": "SflKxwRJSM...",
    "alg": "MDP+ES256+JP", // merkle disclosure with json pointer
    "zip": "DEF"

    "signature": "SflKxwRJSM", // over the entire protected header, like a detached jws, but only the signature part.
  },

Otherwise if we reserved a member “jws” we would have an encoded JWS header, nested insider an encoded JWP header, each with potentially different alg zip kid, etc…

 "protected": {

    "kid": "did:example:123#key-0"

    "root": "SflKxwRJSM...",
    "alg": "MDP+ES256+JP", // merkle disclosure with json pointer
    "zip": "DEF"

    "jws": "{alg, kid, zip}..SflKxwRJSM", // over the entire protected header, but confusing imo...
  },

For merkle proofs a signature over the root must be disclosed, so we need a "signature" in the "protected header".

[OR13]

Another option might be to put the "signature for the merkle root"... on the end of the "proof" object...

So instead of this:

Option 1

{

  // From JWP "Protected Header"
  // always disclosed
  "protected": {

    "kid": "did:example:123#key-0"

    "root": "SflKxwRJSM...",
    "alg": "MDP+ES256+JP", // merkle disclosure with json pointer
    "zip": "DEF"

    "signature": "SflKxwRJSM", // over the entire protected header.
  },

  // From JWP "Payloads" and "Proof"
  // selectively disclosed
  "payloads": [ { message }, ...]
  "proof": [ { path, nonce }, ... ],
}

We would have this:

Option 2

{

  // From JWP "Protected Header"
  // always disclosed
  "protected": {

    "kid": "did:example:123#key-0"

    "root": "SflKxwRJSM...",
    "alg": "MDP+ES256+JP", // merkle disclosure with json pointer
    "zip": "DEF"
  },

  // From JWP "Payloads" and "Proof"
  // selectively disclosed
  "payloads": [ { message }, ...]

  // first part selectively disclosed, second part required.
  "proof": [ { path, nonce}, ... ] + "SflKxwRJSM", // signature over the entire protected header.
}

Downside of this would be constantly splitting the signature from the proofs when deriving.... if thats acceptable, we could also do:

Option 3

{

  // From JWP "Protected Header"
  // always disclosed
  "protected": {
    "kid": "did:example:123#key-0",
    "alg": "MDP+ES256+JP",
    "zip": "DEF"
  },

  // From JWP "Payloads" and "Proof"

  // selectively disclosed
  "payloads": [ { message }, ...]

  // first part selectively disclosed, second part required.
  "proof": [ { path, nonce}, ... ] " + root + "SflKxwRJSM", // signature over the entire protected header.
}

[OR13]

I prefer Option 1

[quartzjer]

While it's not as ideal from a parsing perspective, Option 3 is I believe the correct abstraction.

The challenge is really in the interface to the JWA, the implementing algorithm isn't expected to know or parse the actual protected header, it is encoded before even reaching it (see JWS 5.1).

The only part that the algorithm has full control over is the one that it generates, the signature/proof, and this value is opaque to the application layer and preserved as an encoded byte array to be passed back in for validation un-altered.

The merkle algorithm will need to encode everything it requires both to generate a disclosure and to validate the disclosed into the singular proof value. I think it's completely legit for that to still be a JSON object, but to the higher JWP layer it will be an opaque base64url encoded octet string.

My pseudocode version of your Option 3 would look like:

merkle_proof = {
  "proofs":[ { path, nonce}, ...]
  "root":"SflKxwRJSM...",
  "signature":"ZflKxwRJSM..."
}

proof_b64 = base64url(JSON.stringify(merkle_proof))

message1_b64 = base64url(JSON.stringify({message}))

header = {
  "kid": "did:example:123#key-0",
  "alg": "MDP+ES256+JP",
  "zip": "DEF"
}

header_b64 = base64url(JSON.stringify(merkle_proof))

jwp_json = {
  "protected": header_b64,
  "payloads": [ message1_b64, ...]
  "proof": proof_b64
}

jwp_compact = header_b64 + "." + jwp_json.payloads.join("~") + "." + proof_b64

[OR13]

@quartzjer thanks, this makes a lot of sense, I will try and revise my proposal to align with it better.

[OR13]

@quartzjer I finally got some time to hack this out...

https://github.com/OR13/jwp-mdt/tree/main/ref-impl#json-web-proof-for-merkle-proof-based-selective-disclosure

I opted not to provide an lyt transform for the payloads, since I still don't fully understand how to do that... therefore these proofs can only be used on arrays, not objects.

I like that it works with regular boring JWKs and JWS.
(especially given the limited support in HSM for bls12381 or ed22519)....

I don't love the constant base64url encoding and decoding... or the current header configurations.

I also don't love that I had to invent a compact encoding for merkle proofs in JSON...
would really love to have just used another spec for that.

I think a reasonable next step would be to generalize from arrays of payloads to objects, and start attempting to use lyt...

My naive plan for that is something like this:

const credential = { context, type, issuer, issuanceDate, credentialSubject } 
// no proof since we will be used external proof per the spec

const lyt = generateLayoutFromVc(credential);
const payloads = getPayloadsFromLyt(credential, lyt);
// ... then the rest is the same as it is now ... except `lyt` is embedded in the JWP header.

// On the verify side, we need an option to 
// transform the verified result back to an object... only after verifying...
// this is identical to the JWT behavior, you don't get a verification boolean, 
// you get back an object

const credential = getVcFromJwp(jwp);
// This function needs to use layout with the jwp to construct the vc with disclosed terms.

Here is an example of JWP that only supports array types...

{
  "protected": {
    "kid": "did:example:123#key-0",
    "alg": "MDP+EdDSA",
    "zip": "DEF"
  },
  "payloads": ["YQ", "Yg", "Yw"],
  "proof": "eyJyb290IjoiYzllYjU1YWU4OTk0M2Q0MDRmY2U2MzI3MTM4MTAxYzgwYjIyYzcxYmIzYTY2NmFkZTBiYWY5YTRiZmIzN2JjNiIsIm5vbmNlIjoidXJuOnV1aWQ6ZTJjYWRlMDQtMjg5Ny00YTUwLTk4NDItY2QyZGQ5ZGVhZTEwIiwicHJvb2ZzIjoiZUp5VnpOc1NRa0FBQU5CLzhhcVpWY3VXUitSU2E1bHFpRzE2MEtJaHViUlc2dXY3aHM0SG5JdGtSR1VEcmZaTVN3cEgwbmFkb3BwNTdDSVdlRUh5R2dvODFUalYyWWFNV0JoRjVhSVBxaWZ0cUowMk9lcWhTOEptV0hKTWsva2didThQMFRQZVFmWlVIR2toR1k0UWRtbHoyTis5VVhqeWlFQmY3Unl3cFpWaHpvU3pDQUtWTC8xY2U0Qi9hNnZPdUZ4UXdOOWl6VHI5UlRQUDlhTVkweUJZN1htVWhtMWMyZGJjRHVJclhYOGQra3ZpIiwic2lnbmF0dXJlIjoiQlVVWHZ5VE9LS0hiSHBYUTMydlc5bFJzRVB2WEZsVkF5a3l2YTFIV0pLVUdMTExyV3NwMDN4aVdNWVdOclhweUJRYUhyZHdkWk5CcEp6a0dXSlowQ1EifQ"
}

in the future it would have:

{
  "protected": {
    "kid": "did:example:123#key-0",
    "alg": "MDP+EdDSA",
    "zip": "DEF",
    "lyt": { ... } // and this can be used to return a verified object instead of verified arrays.
  },
  "payloads": ["YQ", "Yg", "Yw"],
  "proof": "eyJyb290IjoiYzllYjU1YWU4OTk0M2Q0MDRmY2U2MzI3MTM4MTAxYzgwYjIyYzcxYmIzYTY2NmFkZTBiYWY5YTRiZmIzN2JjNiIsIm5vbmNlIjoidXJuOnV1aWQ6ZTJjYWRlMDQtMjg5Ny00YTUwLTk4NDItY2QyZGQ5ZGVhZTEwIiwicHJvb2ZzIjoiZUp5VnpOc1NRa0FBQU5CLzhhcVpWY3VXUitSU2E1bHFpRzE2MEtJaHViUlc2dXY3aHM0SG5JdGtSR1VEcmZaTVN3cEgwbmFkb3BwNTdDSVdlRUh5R2dvODFUalYyWWFNV0JoRjVhSVBxaWZ0cUowMk9lcWhTOEptV0hKTWsva2didThQMFRQZVFmWlVIR2toR1k0UWRtbHoyTis5VVhqeWlFQmY3Unl3cFpWaHpvU3pDQUtWTC8xY2U0Qi9hNnZPdUZ4UXdOOWl6VHI5UlRQUDlhTVkweUJZN1htVWhtMWMyZGJjRHVJclhYOGQra3ZpIiwic2lnbmF0dXJlIjoiQlVVWHZ5VE9LS0hiSHBYUTMydlc5bFJzRVB2WEZsVkF5a3l2YTFIV0pLVUdMTExyV3NwMDN4aVdNWVdOclhweUJRYUhyZHdkWk5CcEp6a0dXSlowQ1EifQ"
}

[quartzjer]

There'll be some updates in other issues/threads here about the layout and claims mapping which would make your next step drastically simpler.

I also don't love that I had to invent a compact encoding for merkle proofs in JSON

Yeah, I can definitely see that.

While I know your initial tooling here is JS/TS, if defining an algorithm I would actually lean towards doing it like all other crypto algorithms do in their signature/proof value: packed binary values. The reason it's predominate is it minimizes the attack surface on the format (there is no format to attack) and removes all external dependencies from the essential validation logic.

The proof value here could be the binary octets of root+signature+0x1+path+nonce+0x3+path+nonce... since each of those values is a known fixed size (the 0x1 etc bytes indicate which index the proof is for).

[OR13]

@quartzjer yep, we are already partially doing that here... https://github.com/OR13/jwp-mdt/blob/main/ref-impl/merkle.js#L41

I am familiar with TLV from NFC space, I could easily implement that approach... this is a very good suggestion.

[mprorock]

The proof value here could be the binary octets of root+signature+0x1+path+nonce+0x3+path+nonce... since each of those values is a known fixed size (the 0x1 etc bytes indicate which index the proof is for).

big fan of those kind of well defined data packs - they work well with go and c structs, and are generally very portable and rapid to process

[OR13]

After thinking about this a bit more, you wouldn't want to do this... mostly because path values have a lot of redundant nodes... which are currently eliminated by compression.

We can still get a more compact proof value though, just like this:

root + nonce + signature + compressed_proofs

[OR13]

const base64url = require("base64url");
const { expandProofs } = require("./merkle");
const expandedDerivedJwp = {
  protected: { kid: "did:example:123#key-0", alg: "MDP+EdDSA", zip: "DEF" },
  payloads: ["Yg"],
  proof:
    "eyJyb290IjoiYzllYjU1YWU4OTk0M2Q0MDRmY2U2MzI3MTM4MTAxYzgwYjIyYzcxYmIzYTY2NmFkZTBiYWY5YTRiZmIzN2JjNiIsIm5vbmNlIjoidXJuOnV1aWQ6ZTJjYWRlMDQtMjg5Ny00YTUwLTk4NDItY2QyZGQ5ZGVhZTEwIiwicHJvb2ZzIjoiZUp4RnpMc09nakFVQU5CLzZhb0pscllYNm9aUjFDZytZMkkwRHREYkdud1ZMU2pFK08rNnVaM3A3TjhFYzZjdTFtbWMyWnZTcEV1a0ZpcWpQZ011NFNjWmNDcU5DVU9sZzVScFJnVmtraG5BanRJK3BnWWs4c3lFRW8yZlVrNHBrdmIvWER5c05iOHppcXRxWUFhT0ZjZFJXWTFhSlhoRlBvNjkvaTZQZW5YaTFJWjUzTkVwaXJNWDZYd0lEWnllWWlYV0lVTEJoc244Y3FkdXN0dld5eXA3TllsTW5XWHEyb25KNS9BRk45Zzl3Zz09Iiwic2lnbmF0dXJlIjoiQlVVWHZ5VE9LS0hiSHBYUTMydlc5bFJzRVB2WEZsVkF5a3l2YTFIV0pLVUdMTExyV3NwMDN4aVdNWVdOclhweUJRYUhyZHdkWk5CcEp6a0dXSlowQ1EifQ",
};

const tags = {
  root: "00",
  nonce: "01",
  signature: "02",
  proofs: "03",
};

const alternateProofEncoding = (jwp) => {
  const decodedProof = JSON.parse(base64url.decode(jwp.proof));
  const root = Buffer.from(decodedProof.root, "hex");
  const nonce = Buffer.from(decodedProof.nonce);
  const signature = Buffer.from(decodedProof.signature, "base64");
  const proofs = Buffer.from(decodedProof.proofs, "base64");
  const buff = Buffer.concat([
    Buffer.concat([
      Buffer.from(tags.root, "hex"),
      Buffer.from(root.length.toString(16), "hex"),
      root,
    ]),
    Buffer.concat([
      Buffer.from(tags.nonce, "hex"),
      Buffer.from(nonce.length.toString(16), "hex"),
      nonce,
    ]),
    Buffer.concat([
      Buffer.from(tags.signature, "hex"),
      Buffer.from(signature.length.toString(16), "hex"),
      signature,
    ]),
    Buffer.concat([
      Buffer.from(tags.proofs, "hex"),
      Buffer.from(proofs.length.toString(16), "hex"),
      proofs,
    ]),
  ]);
  return {
    ...jwp,
    proof: base64url.encode(buff),
  };
};

it("encode as TLV", () => {
  console.log(expandedDerivedJwp.proof.length); // 646
  const jwp2 = alternateProofEncoding(expandedDerivedJwp);
  console.log(jwp2.proof.length); // 436
});