diff --git a/docs/getting-started/guides/diamond/diamond.md b/docs/getting-started/guides/diamond/diamond.md index e2997cb..b484700 100644 --- a/docs/getting-started/guides/diamond/diamond.md +++ b/docs/getting-started/guides/diamond/diamond.md @@ -96,6 +96,14 @@ If you want your Diamond to behave like an `ERC20`, `ERC721`, or other popular s DiamondERC165 OpenZeppelin ERC165-based Facet + + DiamondOwnable + OpenZeppelin Ownable-based Facet + + + DiamondAccessControl + OpenZeppelin AccessControl-based Facet + InitializableStorage OpenZeppelin Initializable-based Storage diff --git a/docs/getting-started/guides/libs/data-structures/incremental-merkle-tree.md b/docs/getting-started/guides/libs/data-structures/incremental-merkle-tree.md index 4b7d6f4..a1b9ff2 100644 --- a/docs/getting-started/guides/libs/data-structures/incremental-merkle-tree.md +++ b/docs/getting-started/guides/libs/data-structures/incremental-merkle-tree.md @@ -2,7 +2,7 @@ ## Introduction -The Incremental Merkle Tree library library provides a cost-effective and straightforward method for maintaining the Merkle Tree data structure on-chain, including the capability to retrieve its root at the contract level. It also offers the flexibility to set a custom hash function, such as the Poseidon hash function, to make it ZKP-friendly. It has been implemented following the analysis of the [Deposit Contract Verification](https://github.com/runtimeverification/deposit-contract-verification/blob/master/deposit-contract-verification.pdf). +The Incremental Merkle Tree library provides a cost-effective and straightforward method for maintaining the Merkle Tree data structure on-chain, including the capability to retrieve its root at the contract level. It also offers the flexibility to set a custom hash function, such as the Poseidon hash function, to make it ZKP-friendly. It has been implemented following the analysis of the [Deposit Contract Verification](https://github.com/runtimeverification/deposit-contract-verification/blob/master/deposit-contract-verification.pdf). ## Implementation @@ -76,7 +76,7 @@ function setHeight(AddressIMT storage tree, uint256 height_) internal; This function sets the height of the Merkle Tree. It **reverts** if the provided height is less than or equal to the current Merkle Tree height. After the Tree Height is set, it will no longer grow automatically, and its height will have to be adjusted manually. -**Time complexity** +#### Time complexity Constant. @@ -114,7 +114,7 @@ This function adds a new element to the tree. In the case where the tree has (`2 However, if the tree height was previously set manually, the operation will revert. -**Time complexity** +#### Time complexity `O(log(n))`, where `n` is the number of elements in the tree. @@ -163,7 +163,7 @@ function setHashers( This function sets custom hash functions to be used for Merkle Tree construction. The function will **revert** if the tree already contains at least one leaf. -#### **Time complexity** +#### Time complexity Constant. @@ -206,7 +206,7 @@ function root(AddressIMT storage tree) internal view returns (bytes32); This function calculates and returns the root of the Merkle Tree based on the elements that were added to the tree previously and the tree's height -#### **Time complexity** +#### Time complexity `O(log(n) + h)`, where `n` is the number of elements in the tree and `h` is the height of the tree. @@ -238,7 +238,7 @@ function height(AddressIMT storage tree) internal view returns (uint256) This function returns the tree height, which corresponds directly to the length of the `branches` array. -#### **Time complexity** +#### Time complexity Constant. @@ -276,7 +276,7 @@ function length(AddressIMT storage tree) internal view returns (uint256); This function returns the number of leaves that have been added to the Merkle Tree. -#### **Time complexity** +#### Time complexity Constant. @@ -314,7 +314,7 @@ function isCustomHasherSet(AddressIMT storage tree) internal view returns (bool) This function returns true, if the custom hash function was set, and false otherwise. -#### **Time complexity** +#### Time complexity Constant. diff --git a/docs/getting-started/guides/libs/data-structures/memory-vector.md b/docs/getting-started/guides/libs/data-structures/memory-vector.md index 0c3808b..96296ed 100644 --- a/docs/getting-started/guides/libs/data-structures/memory-vector.md +++ b/docs/getting-started/guides/libs/data-structures/memory-vector.md @@ -138,7 +138,7 @@ function push(AddressVector memory vector, address[] memory values_) internal pu This function appends new elements to the `vector`. If there is no space left, it reallocates memory, doubling capacity. -**Time complexity** +#### Time complexity Amortized `O(1)` when a single `value_` is passed. Otherwise, it is amortized `O(n)`, where `n` is the `values_.length`. @@ -175,7 +175,7 @@ function pop(AddressVector memory vector) internal pure; This function removes the last element from the `vector`. It neither changes the capacity nor reallocates memory. It will revert if called on an empty `vector`. -**Time complexity** +#### Time complexity Constant. @@ -273,7 +273,7 @@ function at( This function returns an element by its 0-based `index_`. It is a read-only function. It will revert if the `index_` is out of bounds. -**Time complexity** +#### Time complexity Constant. @@ -309,7 +309,7 @@ function length(AddressVector memory vector) internal pure returns (uint256); This function returns the length of the `vector`. It's a read-only function. -**Time complexity** +#### Time complexity Constant. @@ -354,7 +354,7 @@ function toArray( This function returns a raw array `vector` based on. It's a read-only function. -**Time complexity** +#### Time complexity Constant, as it's a pointer copy, not a deep copy. diff --git a/docs/getting-started/guides/libs/data-structures/sparse-merkle-tree.md b/docs/getting-started/guides/libs/data-structures/sparse-merkle-tree.md new file mode 100644 index 0000000..ca533d6 --- /dev/null +++ b/docs/getting-started/guides/libs/data-structures/sparse-merkle-tree.md @@ -0,0 +1,596 @@ +# #️⃣ Sparse Merkle Tree + +## Introduction + +The Sparse Merkle Tree library provides a way to efficiently store an entire Merkle Tree data structure on-chain. This includes the capability to obtain **Inclusion** or **Exclusion** Merkle Tree Proofs (MTP) directly from the contract. Additionally, the library provides the flexibility to set a custom hash function, such as the Poseidon hash function, making it ZKP-friendly. Its origin is based on the [iden3 SMTLib implementation](https://github.com/iden3/contracts/blob/master/contracts/lib/SmtLib.sol) and the ["Sparse Merkle Trees" PDF](https://docs.iden3.io/publications/pdfs/Merkle-Tree.pdf), but it has been optimized and polished. + +## Implementation + +The `SparseMerkleTree` library contains three main structures: + +1. **SMT**: This structure stores all nodes, keeps track of their count, contains a pointer to the Root Node, and maintains the tree's depth. +2. **Node**: This structure contains information about the tree element, including its children (left and right), hash, key, value, and type. +3. **Proof Structure**: This includes the Merkle Tree Proof (MTP) and auxiliary information for the MTP proof. + +These data structures store all leaves and build the tree from the top to bottom. The gas cost for addition increases linearly with each tree level. The gas cost growth can be approximated by the following formula: `y = 92,457x + 255,689` + +For example, if a leaf is added at tree level `50`, the addition operation would cost `4,878,539` gas units. + +In the example below, the tree contains three elements. SMT defines three different types of nodes: + +* **Empty**: This means all values are zero. +* **Middle**: A node that does not contain a key and value. +* **Leaf**: A node that contains a key and value, with both child left and right set to zero. + +Each node also contains a hash; for an `Empty` type node, it is zero. + +For a `Middle` node, it is calculated as follows: `H(H_L || H_R)`, where `H` is the hash function used within the library, `H_L` is the hash of the left node, and `H_R` is the hash of the right node. By default, the hash function is `Keccak-256`, but it can be changed to another, such as the `Poseidon` hash function. + +For a `Leaf` node, the hash is calculated like: `H(1 || k || v)`, where `1` acts as a domain separator, `k` is a key that determines where the leaf will be located in the tree, and `v` is a value representing some data. + +
+ +
SMT that contains three elements
+
+ +When a new element is added to the tree, the binary representation of the key is used to determine the correct placement within the tree. The following steps are taken: + +1. The tree is recursively navigated to find an "empty" slot (an `Empty` node) where the new `Leaf` node can be placed. +2. The path is decided based on the element's key, left is chosen for a bit value of 0 and right for a bit value of 1. +3. If a `Leaf` node is encountered during the insertion process, it is pushed down, and `Middle` nodes are inserted in their stead until a bit that differs between the leaves is identified. This scenario is illustrated in the image above. When the differing bit at position 3 leads to the creation of a branch at the tree's third level. + +The aim of the implementation is to optimize gas efficiency for Sparse Merkle Tree operations while allowing flexibility with various types and values of keys. The significant updates have been made as follows: + +* Optimized storage usage to reduce the number of storage slots. +* Added the ability to set custom hash functions. +* Removed methods and associated storage for managing the tree root's history. + +A detailed breakdown of gas usage for the addition of 16,001 leaves (using the addBytes32 method) to a tree of size 80 is provided below: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
StatisticValue
Mean16,001
Mean1,444,220 gas
Std Dev209,147.6 gas
Min177,853 gas
25%1,317,555 gas
50%1,461,562 gas
75%1,554,030 gas
Max2,723,812 gas
+ +## Functions + +To use the `SparseMerkleTree` library, you need to import it. + +```solidity +import "@solarity/solidity-lib/libs/data-structures/SparseMerkleTree.sol"; +``` + +And optionally bind it to the type with the `using` statement. + +```solidity +using SparseMerkleTree for SparseMerkleTree.UintSMT; +using SparseMerkleTree for SparseMerkleTree.Bytes32SMT; +using SparseMerkleTree for SparseMerkleTree.AddressSMT; +``` + +### constructor + +```solidity +function initialize(UintSMT storage tree, uint256 maxDepth_) internal; +``` + +```solidity +function initialize(Bytes32SMT storage tree, uint256 maxDepth_) internal; +``` + +```solidity +function initialize(AddressSMT storage tree, uint256 maxDepth_) internal; +``` + +#### Description + +The function is used for initializing the Sparse Merkle Tree data structure. To start working with a tree, it is required to set a maximum depth. This maximum depth is used in the `getProof` and `getNodeByKey` functions. + +#### Time complexity + +Constant. + +#### Example + +```solidity +SparseMerkleTree.UintSMT public uintTree; + +uintTree.initialize(20); + +// Reverts: SparseMerkleTree: tree is already initialized +uintTree.initialize(25); +``` + +### setMaxDepth + +```solidity +function setMaxDepth(UintSMT storage tree, uint256 maxDepth_) internal; +``` + +```solidity +function setMaxDepth(Bytes32SMT storage tree, uint256 maxDepth_) internal; +``` + +```solidity +function setMaxDepth(AddressSMT storage tree, uint256 maxDepth_) internal; +``` + +#### Description + +This function sets the maximum depth of the Merkle Tree. It reverts if the provided height is less than or equal to the current Merkle Tree depth. Also, it cannot exceed 256 due to the limitations of the uint256 data type; depths greater than 256 are not feasible. Practically, this number is even lower, around 100, due to EVM constraints. During tests using the Hardhat environment, we were not able to reach a depth greater than 96. + +#### Time complexity + +Constant. + +#### Example + +```solidity +SparseMerkleTree.UintSMT public uintTree; + +uintTree.setMaxDepth(2); + +uintTree.getMaxDepth(); // 2 + +// Reverts: SparseMerkleTree: max depth can only be increased +uintTree.setHeight(1); +``` + +### setHashers + +```solidity +function setHashers( + UintSMT storage tree, + function(bytes32, bytes32) view returns (bytes32) hash2_, + function(bytes32, bytes32, bytes32) view returns (bytes32) hash3_ +) internal; +``` + +```solidity +function setHashers( + Bytes32SMT storage tree, + function(bytes32, bytes32) view returns (bytes32) hash2_, + function(bytes32, bytes32, bytes32) view returns (bytes32) hash3_ +) internal; +``` + +```solidity +function setHashers( + AddressSMT storage tree, + function(bytes32, bytes32) view returns (bytes32) hash2_, + function(bytes32, bytes32, bytes32) view returns (bytes32) hash3_ +) internal; +``` + +#### Description + +This function sets custom hash functions to be used for Merkle Tree construction. +The function will **revert** if the tree already contains at least one leaf. + +#### Time complexity + +Constant. + +#### Example + +```solidity +SparseMerkleTree.UintSMT public uintTree; + +function _hash2(bytes32 element1_, bytes32 element2_) internal pure returns (bytes32) { + return bytes32(PoseidonUnit2L.poseidon([uint256(element1_), uint256(element2_)])); +} + +function _hash3( + bytes32 element1_, + bytes32 element2_, + bytes32 element3_ +) internal pure returns (bytes32) { + return + bytes32( + PoseidonUnit3L.poseidon( + [uint256(element1_), uint256(element2_), uint256(element3_)] + ) + ); +} + +uintTree.setHashers(_hash2, _hash3); +``` + +### add + +```solidity +function add(UintSMT storage tree, uint256 key_, uint256 value_) internal; +``` + +```solidity +function add(Bytes32SMT storage tree, bytes32 key_, bytes32 value_) internal; +``` + +```solidity +function add(AddressSMT storage tree, bytes32 key_, address value_) internal' +``` + +#### Description + +This function adds a new element to the tree. The algorithm for adding an element to the tree is recursive; therefore, the maximum depth specifies the point up to which the recursion can proceed. + +#### Time complexity + +`O(n)`, where `n` is the max depth of the tree. + +#### Example + +```solidity +SparseMerkleTree.UintSMT public uintTree; + +uintTree.initialize(20); + +// key: 3; value: 5 +// Leaf hash: H(1 || 3 || 5) +uintTree.add(3, 5); + +// key: 1; value: 10 +// Leaf hash: H(1 || 1 || 10) +uintTree.add(1, 10); + +// 0x03a14b15187328b46952f37dbb8f36620c8f12e97e4c0dc8b147e7060337c2ab +uintTree.getRoot(); +``` + +### root + +```solidity +function getProof(UintSMT storage tree, uint256 key_) internal view returns (Proof memory); +``` + +```solidity +function getProof(Bytes32SMT storage tree, bytes32 key_) internal view returns (Proof memory); +``` + +```solidity +function getProof(AddressSMT storage tree, bytes32 key_) internal view returns (Proof memory); +``` + +#### Description + +This function generates Inclusion/Exclusion proofs for an element in the tree. The process is as follows: + +1. **Algorithm Initiation**: When a key is provided, the algorithm recursively traverses the tree. +2. **Inclusion Proof**: If the algorithm encounters a `Leaf` with exactly the same key, it will return an Inclusion proof, indicating that the element exists within the tree. +3. **Exclusion Proof with Different Key**: If the algorithm encounters a `Leaf` whose key does not match the requested one, it will return an Exclusion proof. This proof includes the data of the encountered leaf, indicating that the requested element does not exist in the tree. +4. **Exclusion Proof with Empty Node**: If the algorithm encounters an empty node, it will also return an Exclusion proof, signifying that the element does not exist in the tree. + +#### Time complexity + +`O(n)`, where `n` is the max depth of the tree. + +#### Example + +```solidity +SparseMerkleTree.UintSMT public uintTree; + +uintTree.add(3, 5); +uintTree.add(1, 10); + +uintTree.getProof(3); +// Root: 0x03a14b15187328b46952f37dbb8f36620c8f12e97e4c0dc8b147e7060337c2ab +// Siblings: [ +// 0x0000000000000000000000000000000000000000000000000000000000000000 +// 0x273b79dd2dbb4163ccbee92a259242d3d728b787e5cb5b69e45509c6b8b6a19c +// 0x0000000000000000000000000000000000000000000000000000000000000000 +// ... (x16) +// 0x0000000000000000000000000000000000000000000000000000000000000000 +// ] +// existence: true +// index: 0x0000000000000000000000000000000000000000000000000000000000000003 +// value: 0x0000000000000000000000000000000000000000000000000000000000000005 +// auxExistence: false +// auxIndex: 0x0000000000000000000000000000000000000000000000000000000000000000 +// auxValue: 0x0000000000000000000000000000000000000000000000000000000000000000 + +uintTree.getProof(7); +// Root: 0x03a14b15187328b46952f37dbb8f36620c8f12e97e4c0dc8b147e7060337c2ab +// Siblings: [ +// 0x0000000000000000000000000000000000000000000000000000000000000000 +// 0x273b79dd2dbb4163ccbee92a259242d3d728b787e5cb5b69e45509c6b8b6a19c +// 0x0000000000000000000000000000000000000000000000000000000000000000 +// ... (x16) +// 0x0000000000000000000000000000000000000000000000000000000000000000 +// ] +// existence: false +// index: 0x0000000000000000000000000000000000000000000000000000000000000007 +// value: 0x0000000000000000000000000000000000000000000000000000000000000005 +// auxExistence: true +// auxIndex: 0x0000000000000000000000000000000000000000000000000000000000000003 +// auxValue: 0x0000000000000000000000000000000000000000000000000000000000000005 + +uintTree.getProof(2); +// Root: 0x03a14b15187328b46952f37dbb8f36620c8f12e97e4c0dc8b147e7060337c2ab +// Siblings: [ +// 0x0000000000000000000000000000000000000000000000000000000000000000 +// 0x273b79dd2dbb4163ccbee92a259242d3d728b787e5cb5b69e45509c6b8b6a19c +// 0x0000000000000000000000000000000000000000000000000000000000000000 +// ... (x16) +// 0x0000000000000000000000000000000000000000000000000000000000000000 +// ] +// existence: false +// index: 0x0000000000000000000000000000000000000000000000000000000000000002 +// value: 0x0000000000000000000000000000000000000000000000000000000000000000 +// auxExistence: false +// auxIndex: 0x0000000000000000000000000000000000000000000000000000000000000000 +// auxValue: 0x0000000000000000000000000000000000000000000000000000000000000000 +``` + +### getRoot + +```solidity +function getRoot(UintSMT storage tree) internal view returns (bytes32); +``` + +```solidity +function getRoot(Bytes32SMT storage tree) internal view returns (bytes32); +``` + +```solidity +function getRoot(AddressSMT storage tree) internal view returns (bytes32); +``` + +#### Description + +This function calculates and returns the root of the Merkle Tree based on the elements that have been previously added to the tree. + +#### Time complexity + +Constant. + +#### Example + +```solidity +SparseMerkleTree.UintSMT public uintTree; + +uintTree.add(3, 5); +uintTree.add(1, 10); + +// 0x03a14b15187328b46952f37dbb8f36620c8f12e97e4c0dc8b147e7060337c2ab +uintTree.getRoot(); +``` + +### getNode + +```solidity +function getNode(UintSMT storage tree, uint256 nodeId_) internal view returns (Node memory); +``` + +```solidity +function getNode( + Bytes32SMT storage tree, + uint256 nodeId_ +) internal view returns (Node memory); +``` + +```solidity +function getNode( + AddressSMT storage tree, + uint256 nodeId_ +) internal view returns (Node memory); +``` + +#### Description + +This function returns a node element by its index in the tree. Each `Middle` or `Leaf` element is assigned a sequential or numerical order when added to the tree, stqarting with index 1. + +#### Time complexity + +Constant. + +#### Example + +```solidity +SparseMerkleTree.UintSMT public uintTree; + +uintTree.add(3, 5); +uintTree.add(1, 10); + +uintTree.getNode(1); // key: 3; value: 5; type: Leaf +uintTree.getNode(2); // key: 1; value: 10; type: Leaf +uintTree.getNode(3); // childLeft: 2; childRight: 1; type: Middle +``` + +### getNodeByKey + +```solidity +function getNodeByKey(UintSMT storage tree, uint256 key_) internal view returns (Node memory); +``` + +```solidity +function getNodeByKey( + Bytes32SMT storage tree, + bytes32 key_ +) internal view returns (Node memory); +``` + +```solidity +function getNodeByKey( + AddressSMT storage tree, + bytes32 key_ +) internal view returns (Node memory); +``` + +#### Description + +This function returns a `Leaf` node for the given key. + +#### Time complexity + +`O(n)`, where `n` is the max depth of the tree. + +#### Example + +```solidity +SparseMerkleTree.UintSMT public uintTree; + +uintTree.add(3, 5); +uintTree.add(1, 10); + +uintTree.getNodeByKey(3); // key: 3; value: 5; +uintTree.getNodeByKey(10); // key: 0; value: 0; +``` + +### getMaxDepth + +```solidity +function getMaxDepth(UintSMT storage tree) internal view returns (uint256); +``` + +```solidity +function getMaxDepth(Bytes32SMT storage tree) internal view returns (uint256); +``` + +```solidity +function getMaxDepth(AddressSMT storage tree) internal view returns (uint256) +``` + +#### Description + +This function returns the maximum depth of the tree. + +#### Time complexity + +Constant. + +#### Example + +```solidity +SparseMerkleTree.UintSMT public uintTree; + +uintTree.getMaxDepth(); // 0 + +uintTree.setMaxDepth(20); + +uintTree.getMaxDepth(); // 20 +``` + +### getNodesCount + +```solidity +function getNodesCount(UintSMT storage tree) internal view returns (uint256); +``` + +```solidity +function getNodesCount(Bytes32SMT storage tree) internal view returns (uint256); +``` + +```solidity +function getNodesCount(AddressSMT storage tree) internal view returns (uint256); +``` + +#### Description + +This function returns the total number of nodes (the sum of the Middle and Leaf nodes) in the Merkle Tree. + +#### Time complexity + +Constant. + +#### Example + +```solidity +SparseMerkleTree.UintSMT public uintTree; + +uintTree.add(3, 5); + +uintTree.getNodesCount(); // 1; 1 Leaf node + +uintTree.add(1, 10); + +uintTree.getNodesCount(); // 3; 2 Leaf nodes and 1 Middle +``` + +### isCustomHasherSet + +```solidity +function isCustomHasherSet(UintSMT storage tree) internal view returns (bool); +``` + +```solidity +function isCustomHasherSet(Bytes32SMT storage tree) internal view returns (bool); +``` + +```solidity +function isCustomHasherSet(AddressSMT storage tree) internal view returns (bool); +``` + +#### Description + +This function returns true, if the custom hash function was set, and false otherwise. + +#### Time complexity + +Constant. + +#### Example + +```solidity +SparseMerkleTree.UintSMT public uintTree; + +function _hash2(bytes32 element1_, bytes32 element2_) internal pure returns (bytes32) { + return bytes32(PoseidonUnit2L.poseidon([uint256(element1_), uint256(element2_)])); +} + +function _hash3( + bytes32 element1_, + bytes32 element2_, + bytes32 element3_ +) internal pure returns (bytes32) { + return + bytes32( + PoseidonUnit3L.poseidon( + [uint256(element1_), uint256(element2_), uint256(element3_)] + ) + ); +} + +uintTree.isCustomHasherSet(); // false + +uintTree.setHashers(_hash2, _hash3); + +uintTree.isCustomHasherSet(); // true +``` + +## Production References + +* [rarimo/voting-contracts](https://github.com/rarimo/voting-contracts) uses SMT to store commitments of users registrations. diff --git a/static/img/docs/sparse-merkle-tree.png b/static/img/docs/sparse-merkle-tree.png new file mode 100644 index 0000000..b73364d Binary files /dev/null and b/static/img/docs/sparse-merkle-tree.png differ