Introduce SparseMerkleTree trait

[plafer]

Work towards: #243

Left to do:

• find proper naming for SparseMerkleTree trait and NewSmt struct
• tests
• double check trait bounds on associated types for SparseMerkleTree, and double check which arguments should be passed by move vs reference
• Confirm: Most significant Felt of a Word is the last Felt of the Word
• (created issue) Smt::Opening: Change to a struct close to TieredSmtProof for Smt
• SparseMerkleTree::open(): take a reference to key (after key_to_index() change)
• Consider making get_leaf() -> &Leaf (instead of Leaf)

[bobbinth]

Looks good! Thank you! I reviewed most things excluding tests and left some comments inline. Overall, I think this will work very well. One of the main questions is whether we should expose SparseMerkleTree publicly - my current thinking is that probably not.

In terms of naming, I think SparseMerkleTree is fine for the trait. The NewSmt, I'd probably rename into Smt (and keep SimpleSmt as is). Another option is FullSmt, but I like just Smt better.

We could also come up with a more creative name (Octopus? :)) - but I'm not sure that's needed.

[bobbinth]

I think exposing mutator methods publicly is not ideal because it allows modifying the underlying data structure in invalid ways (e.g., set an invalid root). I think we have two options here:

1. Make the trait private and expose provided methods via pass through methods on implementing structs.
2. Split the trait into two traits and make the one with mutator methods internal. The public trait would then depend on the internal trait.

It seems to me that the first approach is better because:

1. It makes the structures easier to use - i.e., we don't need to import extra traits to get access to a subset of methods on a given struct.
2. It provides more flexibly for the future - i.e., if the implementations of SimpleSmt and NewSmt need to diverge in the future, this would be transparent to the users of the crate.

[bobbinth]

We'd also need to implement something like to_elements() here - e.g.:

pub fn to_elements(&self) -> Vec<Felt>

pub fn into_elements(self) -> Vec<Felt>

We will probably need these to inject the SMT into the advice provider.

[plafer]

Done.

Any reason why we shouldn't implement From for that, instead of custom methods?

[bobbinth]

No strong reason. Sometimes it make the code more readable, but in this case probably doesn't make a difference.

[bobbinth]

There is a special case we need to handle here: value == EMPTY_VALUE. In this case, we need to return the tree into the state it was as if the key wasn't inserted in the first place. This implies:

1. For a single leaf - remove the leaf.
2. For multiple leaf:
   a. If there are more than 2 entries in the leaf - remove the entry associated with the key.
   b. If there are exactly 2 entries, remove the relevant entry and convert the leaf into a single leaf.

I would probably create a couple of helper functions to handle these cases separately (would help with readability).

[plafer]

I handled the case, but was able to make perform_remove() clean. For insert(), I moved it to SmtLeaf; but you can't do that for remove(), since in one case, "removing the leaf" involves removing the element from Smt.leaves.

I'll write the tests, and then find a better way to clean it up

[bobbinth]

How about something like this:

fn perform_remove(&mut self, key: SmtKey) -> Option<Word> {
    let leaf_index: LeafIndex<SMT_DEPTH> = key.into();

    if let Some(leaf) = self.leaves.get_mut(&leaf_index.value()) {
        let (old_value, is_empty) leaf.remove(key);
        if is_empty {
            self.leaves.remove(&leaf_index.value());
        }
        old_value
    }  else {
        // there's nothing stored at the leaf; nothing to update
        None
    }
}

[plafer]

done!

[bobbinth]

If we keep SparseMerkleTree private in this create, we probably won't need to make InnerNode and LeafIndex public either.

[plafer]

Made SparseMerkleTree private, but let LeafIndex public, since LeafIndex: is now part of SimpleSmt's public API (e.g. get_leaf())

[bobbinth]

Looks good! Thank you! I left a few comments inline - all are pretty minor.

Some more general thoughts:

In terms of overall module layout. I wonder if we should do something like this:

merkle
├── smt
│   ├── full
│   │   ├── mod.rs [this is where Smt struct could go]
│   │   └── tests.rs
│   ├── mod.rs [this is where SparseMerkleTree trait would go]
│   └── simple
│       ├── mod.rs [this is where SimpleSmt struct could go]
│       └── tests.rs

The above would be similar to the organization of the Mmr module.

Next, I think the Smt struct is missing several methods (most of these could probably be added in a subsequent PR):

• get_value(&self, key: RpoDigest) -> Word.
• Something similar to Tsmt::prove().
• inner_nodes(&self) -> impl Iterator<Item = InnerNodeInfo>.
• leaves(&self) -> impl Iterator<Item = (u64, &SmtLeaf)>.
• entries(&self) -> impl Iterator<Item = &(RpoDigest, Word)>.

Looking through various Merkle structs, I'm wondering if we should adapt a slightly different naming/method convention for SparseMerkleTree. Specifically:

• We could rename get_leaf_path() -> MerklePath into open() -> (MerklePath, Self::Leaf). Or maybe have a separate Opening associated type. This would be more consistent with the terminology used in crypto papers and similar to what we do in the Mmr module.
• Similarly, maybe we should rename update_leaf() into insert(). The justification is that the operation the user cares about is inserting a value into the underlying tree. Whether it updates a leaf or something else is an implementation detail.

Lastly, this PR is getting pretty big. I wonder if we should split it into two:

1. PR introducing SparseMerkleTree and applying it to SimpleSmt. This would be a breaking PR as it will break things in miden vm/base/node, and maybe client.
2. Once the above PR is merged and updates propagated to downstream repos, we would do another PR to introduce Smt struct.

[plafer]

@bobbinth ready for re-review!

[bobbinth]

Looks good! Thank you! I left a few comments inline - most of the are small nits.

Also, as I mentioned in one of the comments, we should probably remove Smt implementation from this PR and add it in a subsequent PR which also addresses #251.

Lastly, I wouldn't merge this PR just yet. Let's create a PR in Miden VM which incorporates this new SimpleSmt struct - so that once we merge this PR we can also merge that PR and nothing in Miden VM would be broken. After that, we'd make corresponding updates to miden-base and miden-node, and then come back here to finish Smt and remove TieredSmt in the next PR.

[plafer]

Removed Smt, and created #254

[sonarcloud]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

[hackaugusto]

This broke downstream, do we have PRs to fix it on the works?

[bobbinth]

Yes - all fixing PRs have been merged. If you rebase from main and do cargo update it should fix the node build.