Replies: 9 comments 15 replies
-
|
Thanks for posting this @Tuditi, I am looking forward to peoples thoughts and suggestions 😄 |
Beta Was this translation helpful? Give feedback.
-
I consider this critical to enable central outputs that can be used to represent/anchor multiple applications. |
Beta Was this translation helpful? Give feedback.
-
Why don't you just set the metadata as JSON encoded string and put the different pieces of hex encoded data in the JSON level representation? You can define whatever application level JSON schema standard for DIDs or other use cases. |
Beta Was this translation helpful? Give feedback.
-
Could you elaborate who the "we" is in that scenario? I assume the protocol will not inspect / validate those fields. Is this concerning applications that would want to display the meta data without interpreting it like wallets and explorers? Interpreting the data would in many cases require the actual business logic of the specific application, I assume. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, it concerns the higher level applications in general that interpret data according to this standard. The TIP doesn't make any assumptions on the actual business logic, which I also think is out-of-scope. An interesting question is whether there are security risks present if applications handle the metadata as 'trusted'. I think the answer to this question depends on the given application and it's important to disclaim that the fields are not validated. |
Beta Was this translation helpful? Give feedback.
-
From the perspective of the DID method ( There are similar consideration to L2 SC chains regarding backward compatibility though. We can help implementers transition their documents through the library, but to not break existing documents (not under the implementers control) the library would need to support the current and the proposed encoding for the foreseeable future. |
Beta Was this translation helpful? Give feedback.
-
|
What is the current encoding for L2 SC chains? |
Beta Was this translation helpful? Give feedback.
-
|
Are there alternatives to HEX encoding as the meta data field are size constrained and the overhead might be prohibitiv for larger data chunks or co-existance/partitioning. |
Beta Was this translation helpful? Give feedback.
-
Which then would it make less efficient again, no? |
Beta Was this translation helpful? Give feedback.
-
|
In the end the hex representation is just used to denote the bytes. So it's not that we encode the UTF-8 string of the hex encoding of the bytes. Maybe the following line regarding layer 0 should be changed from:
to:
What do you think? |
Beta Was this translation helpful? Give feedback.
-
I think this should be clarified a bit, there is also |
Beta Was this translation helpful? Give feedback.
-
|
I edited it. Is it more clear like this? In the end level 0 doesn't know to much about what the metadata really represents. It just states that it's using hexadecimals to represent whatever else is following. |
Beta Was this translation helpful? Give feedback.
-
|
To me it sounds better now 👍 |
Beta Was this translation helpful? Give feedback.
-
Can you clarify whether this refers to the Metadata Feature that can be put in outputs or the |
Beta Was this translation helpful? Give feedback.
-
|
It refers primarily to the Metadata Feature, but I see no reason why this couldn't be extended to the |
Beta Was this translation helpful? Give feedback.
-
|
Why |
Beta Was this translation helpful? Give feedback.
-
|
Metadata Encoding Standard is the name of the TIP & 1 is the version. Each TIP has to have a name that describes what it does. If this becomes accepted, the community can refer to it as TIP-38. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah it will be just called "TIP-XX: Metadata Encoding Standard". |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for creating this TIP! I have a few questions/comments.
I think the TIP should be more specific and define whether this applies to the bytes in a Metadata Feature or the
Both Metadata Feature and the
If I understand correctly, this TIP would standardize this schema, so could you add the JSON schema that specifies what the types of the {
"standard": "iota-did",
"version": "0.7",
"document": {
"id": "did:iota:0xabc..."
}
}or does it have to be in a specific key, say {
"standard": "iota-did",
"version": "0.7",
"data": {
"document": {
"id": "did:iota:0xabc..."
}
}
} |
Beta Was this translation helpful? Give feedback.
-
|
Hey Philipp, Since the With regard to the layer 0 encoding. This has been a pain point raised before by @eike-hass & @Thoralf-M , so I removed the references to layer 0 encoding. We were confusing the binary representation with the binary data itself. I'll add a JSON Schema for the layer 3 encoding of JSON Objects. New standards should be free to add their own top-level keys in my opinion. Thanks a lot for your thorough review and let me know what you think of the changes! |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for addressing those points, much appreciated! I think the hex encoding is still part of the "Encoding bytes" section, I suppose that one has to be removed as well, then?
I think it would suffice to say:
The Metadata Feature can be part of a Basic, NFT and Alias Outputs within the Feature or (for the latter two) also the Immutable Features section. The way it's written now would imply it only applies to the Feature in the Basic Output, which I assume is not what you want to restrict it to. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, you're right. I updated the missing parts. |
Beta Was this translation helpful? Give feedback.
-
Hey @Tuditi , have you talked with the ISC team about this? While I get that it makes sense to be able to parse the content of a request, I believe ISC uses custom minimal encoding in requests in order to save space and therefore deposits as well. If you are an application (Firefly, Explorer, etc.) and want to display request data, you'll be able to infer whether a piece of bytes should be parsed as an ISC request or not by simply looking at the target
IMO it doesn't make sense. ISC state in the
Through the usual TIP process. In order for such an application level TIP to be successful, you need to demonstrate that it has a working implementation (in this case open source lib for working with MES encoding/decoding, preferably in more languages), furthermore that there is application level support for it. A quote from TIP-1: "It is your responsibility as a TIP author to build community consensus around your idea. Involve as many people in the discussion as you can. Use social media platforms, Discord or Reddit to raise awareness of your idea." "TIPs that have broad support are much more likely to make progress than those that don't receive any comments. Feel free to reach out to the TIP editors in particular to get help to identify stakeholders and obstacles." In this case I'd also ask for clarification with the ISC team and whether anything needs to change on their side with metadata handling. I'd also talk to NFT projects in the ecosystem just so they know that this is compatible with how things are today. You already involved DID people, that's great!
You should, and also libs to help others work with it. Since this is application level, a bare minimum would be a TypeScript SDK.
My understanding is that since you reserved the first possible characters of JSON on layer 1, it's probably not a JSON if you can't detect the encoding. I'd just treat it as plain simple bytes and omit the handling/further processing.
I'd say treat DID as JSON objects, why would you want to develop a special encoding just for one use case? +1 bonus question: Why did you consider only CBOR? Did you look into other commonly used web3 encodings? How about RLP, Borsh or BCS? |
Beta Was this translation helpful? Give feedback.
-
TIP - Metadata Encoding Standard v1 (MES-1)
Preamble
tip: ?
title: Metadata Encoding Standard v1 (MES-1)
description:
author: Nicole O'Brien (@nicole-obrien) [email protected], David De Troch (@Tuditi) [email protected], Matthew Maxwell (@maxwellmattryan) [email protected]
status: Draft
type: Process
created: 2023-05-12
Abstract
The TIP describes the Metadata Encoding Standard v1 (MES-1), a standard for encoding and decoding data in the metadata field of an output. The lack of a standard for metadata encoding/decoding poses a problem for applications that interpret metadata of an output/transaction across the ecosystem. The specification outlines a layered approach to encoding, with each encoding level being a layer of the onion. Level 1 determines the encoding of the data type, with backwards compatibility for existing metadata that uses JSON. Level 2 defines the format of the data. Further levels of encoding can be defined for specific standards. The specification also proposes a flexible system for defining the bytes that tell us how to decode the data, which allows to add future standards. The proposed encoding uses the first bit as a flag and the remaining bits to denote the encoding type.
Motivation
Currently, applications don't have a standard that helps them encode and decode the data fields of a Transaction or Output (e.g. the metadata field of an Alias). Each application is free to define data objects as they see fit, even if the data format already adheres to an RFC standard (e.g. JSON, UTF-8, CBOR, ...). This poses a problem for applications that interpret metadata across the ecosystem. We think it's beneficial to introduce a TIP that standardizes encoding metadata.
Encoding / Decoding
This TIP defines several levels of encoding, much like the layers of an onion. Each layer has to be decoded in order to understand the full message. For simplicity we will order the layers in the order required to decode and read the data at the center of the onion.
This TIP refers to the State Metadata field of an Alias Output and the Metadata Feature as "metadata" throughout the document. TIP-18 defines the byte format of the metadata on a lower level.
Layer 1 - Encoding
Layer 1 is used to determine how we should decode the metadata e.g. should we treat the data as a byte array or a UTF8 string or something else entirely.
Initial sensible layer 1 encodings can be defined as:
Layer 2 - Format
Layer 2 determines the format of the data that the application wants to read. This could represent an existing data or file format or a new custom proprietary one that can be linked in this TIP e.g. JSON, CBOR, XML.
Byte Array - 00
Text Format - 08, 10, 12, 20
Layer 3 - Standard
Layer 3 is an optional standard that can be defined to allow programs to understand the data. Decoding the standards will be specific to the data format chosen. E.g IRC30, defined in TIP-30, contains the definition of how to store metadata of a native token and currently only contains a definition for JSON format but it could be extended to include a decoding for CBOR etc.
JSON
In JSON a standard can be defined using the key
standardand an optional version defined using the keyversion. The JSON scheme at the end of this document provides information around its structure. Currently we have two defined standards for JSON formatted metadata:ISC
Combinations
We also have the ability to shorten encodings to fewer bytes by defining a specific byte for a combination of the above encodings. And in specific cases as we will describe later for backwards compatibility the the byte itself could form part of the data to be parsed.
Encoding bytes
The metadata is represented by binary data, so bytes are a logical choice for markers or flags to decode the data. Therefore we introduce a system to define the bytes which tell us how to decode the data.
Extendable bytes
The system is flexible so new standards can be added in the future. The proposed encoding uses the first bit as a flag and the remaining bits to denote the encoding type. If the flag bit is unset (flag bit = 0), only a single byte signals the format decoding type. Otherwise (flag bit = 1), the decoding also considers the next byte for decoding the format.
This approach provides 128 single byte values to define standards for. An additional 32,760 double byte values ensure that the encoding is future proof.
Rationale
Currently, two application level metadata standards exist: the IRC27 and IRC30 metadata standards for NFTs and Foundries. They don't specify how they are encoded, but applications assume that the metadata field consists of hex encoded JSON objects.
Soon we will introduce three more metadata standards. These relate to the anchoring of smart contract state and DID records and the interaction with the smart contract chains on layer 1. To our knowledge the two smart contract standards save space through proprietary encoding. (There is no need to store a key value pair / object if you know all the fields and can enforce the size and order).
These will not be the only metadata standards introduced in IOTA's lifetime, and applications should decode the metadata without using trial and error. Therefore, we propose a metadata encoding standard before adding any more functionality.
This design represents an efficient encoding that optimizes the space used on the Tangle, but allows future additions to the standard. The standard is backwards compatible with the basic JSON metadata to minimize changes for the TIPs concerning token foundries or NFTs.
Backwards Compatibility
Currently there is no standard for encoding metadata, and each individual standard, such as IRC-27 and IRC-30 states how they encode the metadata field. By introducing this encoding standard, previous stored metadata may contradict the standard.
MES-1 covers the current known metadata standards on the Shimmer network. The only two published standards, to the best of our knowledge, are:
The analysis of these standards resulted in exceptions to the layered encoding approach while also minimizing the bytes used for more common standards.
IRC27 and IRC30 metadata use UTF8/JSON encoding to represent objects in JSON. The JSON standard determines what the initial characters can be. MES-1 supports JSON objects, arrays and strings (i.e., objects are initialized by
{, arrays by[and strings by"). The layer 1 encoding reserves the hex value of these starting characters to ensure maximum backwards compatibility.{is represented by123in UTF8, therefore the hex code reserved is7B[is represented by91in UTF8, therefore the hex code reserved is5B"is represented by34in UTF8, therefore the hex code reserved is22Examples
Byte Array
Metadata:
[01010100, 00000110, 01010111, 10000110, 00010110, 11010111, 00000110, 11000110, 01010010, 11100110, 00110110, 11110110, 11010010, 00100010, 11000000, 10100010, 00000010, 00001111]HEFS
Encoded Metadata
0x00005406578616d706c652e636f6d222c0a2020fUTF-8 String
Metadata
'Hello World'HEFS
Encoded Metadata
0x080048656c6c6f20576f726c64JSON Object
Metadata
{ "id": "12345", "name": "John Doe", "age": 30, "email": "[email protected]", "address": { "street": "123 Main St", "city": "Anytown", "state": "CA", "zip": "12345" } }HEFS
Encoded Metadata
Results in the following metadata:
0x7b0a2020226964223a20223132333435222c0a2020226e616d65223a20224a6f686e20446f65222c0a202022616765223a2033302c0a202022656d61696c223a20226a6f686e646f65406578616d706c652e636f6d222c0a20202261646472657373223a207b0a2020202022737472656574223a2022313233204d61696e205374222c0a202020202263697479223a2022416e79746f776e222c0a20202020227374617465223a20224341222c0a20202020227a6970223a20223132333435220a20207d0a7dIRC-30 Token Standard
Metadata
{ "standard": "IRC30", "name": "Metadata Token", "symbol": "MES1", "decimals": 0 }HEFS
Encoded Metadata
0x7b227374616e64617264223a224952433330222c226e616d65223a224d6574616461746120546f6b656e222c2273796d626f6c223a224d455331222c22646563696d616c73223a307dIota Smart Contract On Chain Request
Metadata
{ "senderContract": 0, "targetContract": 1011572226, "entryPoint": 603251617, "params": { "Items": [ { "key": "0x61", "value": "0x03cbcd6d8659ed1998a452335ae53904dc0af1c99b" } ] }, "allowance": { "baseTokens": 25000000, "nativeTokens": [], "nfts": [] }, "gasBudget": 500000 }HEFS
Encoded Metadata
0x00010100000000025e4b3ca1e3f42320a1070000000000010000000100611500000003cbcd6d8659ed1998a452335ae53904dc0af1c99b0040787d0100000000020000000000JSON Schema for Layer-3 standards
{ "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://github.com/iotaledger/tips/main/tips/TIP-0038/irc38.schema.json", "title": "JSON Schema to define layer-3 JSON standards", "description": "JSON Schema to define layer-3 JSON standards", "type": "object", "properties": { "standard": { "type": "string", "description": "identifier for the metadata standard used", }, "version": { "type": "string", "description": "version of the standard used", "pattern": "^v\\d+.\\d+$" } }, "required": [ "standard" ] }Questions
Beta Was this translation helpful? Give feedback.
All reactions