Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce SparseMerkleTree trait #245

Merged
merged 117 commits into from
Jan 18, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
117 commits
Select commit Hold shift + click to select a range
d4ed8c1
smt: scaffolding
plafer Jan 10, 2024
15544f8
implement `update_leaf_at`
plafer Jan 10, 2024
74b4db6
docstring
plafer Jan 10, 2024
fba526f
move `recompute_nodes_from_index_to_root` into trait
plafer Jan 10, 2024
8f485b9
implement `get_merkle_path`
plafer Jan 10, 2024
78fe6a7
introduce `DEPTH` const generic to remove runtime errors
plafer Jan 10, 2024
d42b38b
leaf index at depth 64
plafer Jan 10, 2024
babd406
add `depth()` method to smt
plafer Jan 10, 2024
adde943
file separators
plafer Jan 10, 2024
8610986
remove hash_value
plafer Jan 10, 2024
445f802
use Vec
plafer Jan 10, 2024
4ec228f
use `InnerNode` in `SimpleSmt`
plafer Jan 10, 2024
d5cee9f
SimpleSmt: make DEPTH const generic
plafer Jan 10, 2024
781ea71
LeafIndex: derives
plafer Jan 10, 2024
8e0cde0
implement SparseMerkleTree for SimpleSmt
plafer Jan 10, 2024
da9ada4
Remove old comment
plafer Jan 10, 2024
f8720d5
Rename get_merkle_path
plafer Jan 10, 2024
eeba178
Remove `SimpleSmt::get_path()` and `get_leaf_path()`
plafer Jan 11, 2024
f5acf5d
add tryfrom NodeIndex for LeafIndex
plafer Jan 11, 2024
f762d8a
impl get_leaf_at on its own
plafer Jan 11, 2024
8688661
impl get_node in terms of get_leaf_at
plafer Jan 11, 2024
b025d48
Remove `get_leaf()`
plafer Jan 11, 2024
9810428
rename get_leaf_at
plafer Jan 11, 2024
0db21a4
remove get_branch_node
plafer Jan 11, 2024
9e08ccd
remove insert_branch_node
plafer Jan 11, 2024
47a34f8
remove duplicate recompute_nodes_from_index_to_root
plafer Jan 11, 2024
dcd8e26
remove private get_leaf_node
plafer Jan 11, 2024
79ffcd8
remove trivial helper
plafer Jan 11, 2024
2c6e38d
remove extra `root()` method
plafer Jan 11, 2024
b661312
add vscode gitignore
plafer Jan 11, 2024
740a316
remove old `get_leaf()`
plafer Jan 11, 2024
66f7330
Rename `update_leaf_at`
plafer Jan 11, 2024
20a623a
add comment
plafer Jan 11, 2024
fba5577
import `Vec` for `no_std`
plafer Jan 11, 2024
11e44c8
fix docstring
plafer Jan 11, 2024
f7872c1
nightly fmt
plafer Jan 11, 2024
b13ef60
new_smt scaffolding
plafer Jan 11, 2024
c1902d7
implement `hash_leaf()`
plafer Jan 11, 2024
dfc3f77
implement `get_leaf()`
plafer Jan 11, 2024
561271b
use `Vec`
plafer Jan 11, 2024
a07e42c
insert_leaf_node scaffold
plafer Jan 11, 2024
bec00e0
insert on leaf with single element
plafer Jan 11, 2024
fb49a69
cmp just keys
plafer Jan 11, 2024
c07d814
insert leaf: multiple
plafer Jan 11, 2024
ac96e5e
reorg file
plafer Jan 11, 2024
9425ea0
no_std vec
plafer Jan 11, 2024
53fe96d
rename `SMT_MAX_DEPTH`
plafer Jan 11, 2024
ae04c79
use `SMT_MAX_DEPTH`
plafer Jan 11, 2024
b9b9d1e
`SMT_MIN_DEPTH`
plafer Jan 11, 2024
9fb9a20
Enforce `SMT_MIN_DEPTH` in `LeafIndex`
plafer Jan 11, 2024
82c84bb
nightly fmt
plafer Jan 11, 2024
0d1fa1c
`as_int` instead of `inner()`
plafer Jan 11, 2024
feabd0f
rename `insert_leaf_node`
plafer Jan 11, 2024
8c6ae9f
add tests mod
plafer Jan 11, 2024
6bf5f07
most signifcant felt: last felt of word
plafer Jan 11, 2024
0ae3464
fix comments
plafer Jan 11, 2024
6c9b87d
`NewSmt` constructor
plafer Jan 11, 2024
795ff96
fix comment
plafer Jan 11, 2024
ff99639
Remove `max_num_entries` in `NewSmt`
plafer Jan 12, 2024
9514ecf
Rename `NewSmt` to `Smt`
plafer Jan 12, 2024
e7899b3
Rename `NewSmt` to `Smt`, and `smt` module to `sparse_merkle_tree`
plafer Jan 12, 2024
a7fdaf9
Replace `Value`'s `Default` trait bound with `EMPTY_VALUE` associated…
plafer Jan 12, 2024
d4c51b8
remove `SimpleSmt`'s `EMPTY_VALUE` constant
plafer Jan 12, 2024
7f8e178
ABSTRACT -> PROVIDED
plafer Jan 12, 2024
e93c9a2
remove `depth()` method
plafer Jan 12, 2024
fe6a9dc
`SmtKey`: use `RpoDigest` internally
plafer Jan 12, 2024
306dad1
add constructors to `SmtKey`
plafer Jan 12, 2024
f8e474f
optimize `SmtLeaf::hash()`
plafer Jan 12, 2024
21220a4
impl `to_elements()` and `into_elements()` for `SmtLeaf`
plafer Jan 12, 2024
8ebafed
Make `SparseMerkleTree` pub(crate)
plafer Jan 12, 2024
b0e0bdd
add sections to `Smt`
plafer Jan 12, 2024
662e886
remove `SimpleSmt::depth()`
plafer Jan 12, 2024
59a037a
fix `get_inner_node()` docstring
plafer Jan 12, 2024
05e3271
remove `get_inner_node` from public API
plafer Jan 12, 2024
19dde05
insert_value: if no value at key, always return `None`
plafer Jan 12, 2024
5f17771
cover case of inserting an `EMPTY_VALUE`
plafer Jan 12, 2024
8ad9d80
`NewLeaf::insert`
plafer Jan 12, 2024
97f06ab
update comment
plafer Jan 12, 2024
29768fb
test_smt_insert_at_same_key
plafer Jan 12, 2024
52cbc21
test_smt_insert_at_same_key_2
plafer Jan 12, 2024
132beaf
fix most_significant_felt for `SmtKey`
plafer Jan 12, 2024
31e26bf
fix test_smt_insert_at_same_key_2
plafer Jan 12, 2024
8bd01b9
test inserting multiple values
plafer Jan 12, 2024
bc1dca9
test_smt_insert_multiple_values
plafer Jan 12, 2024
e95af3f
simplify interface
plafer Jan 12, 2024
e8566e0
core::iter instead of std
plafer Jan 12, 2024
69c1959
test_smt_path_to_keys_in_same_leaf_are_equal
plafer Jan 12, 2024
986b14d
cleanup `perform_remove()`
plafer Jan 12, 2024
5e5c418
clippy
plafer Jan 12, 2024
2802182
fix comment
plafer Jan 16, 2024
cca0760
add `depth()` function
plafer Jan 16, 2024
8fce44c
add depth() fn to simple smt
plafer Jan 16, 2024
8510007
`SparseMerkleTtree`: rename `update_leaf` to `insert`
plafer Jan 16, 2024
e162242
`Smt`: rename `update_leaf`
plafer Jan 16, 2024
1265dea
`SimpleSmt`: rename `update_leaf`
plafer Jan 16, 2024
7bc8eb0
fmt
plafer Jan 16, 2024
4b2f3c5
get_leaf_path -> open
plafer Jan 16, 2024
40c3b76
`SparseMerkleTree::Opening` type
plafer Jan 16, 2024
fd60769
`open()` returns `Self::Opening`
plafer Jan 16, 2024
9d683b6
Introduce `key_to_leaf_index`
plafer Jan 16, 2024
3ba0bb0
open() takes a reference to `key`
plafer Jan 16, 2024
4eee502
new directory structure
plafer Jan 16, 2024
7532d72
`SimpleSmtTrait`: Return possibly borrowed data
plafer Jan 16, 2024
3d583d0
Revert "`SimpleSmtTrait`: Return possibly borrowed data"
plafer Jan 17, 2024
5ff055f
docstring
plafer Jan 17, 2024
9c7a829
fix `open()` docstring
plafer Jan 17, 2024
771ce46
move `depth()` function up
plafer Jan 17, 2024
a6a0ddc
fix var name
plafer Jan 17, 2024
e4d9c5a
move `open()` down
plafer Jan 17, 2024
eb24376
fix comment
plafer Jan 17, 2024
9dc0228
`ValuePath`: adjust constructor
plafer Jan 17, 2024
0e5c64d
Use `ValuePath` in simple smt
plafer Jan 17, 2024
5d76006
remove `Smt`
plafer Jan 17, 2024
1c92eeb
make `new_max_depth()` const
plafer Jan 17, 2024
b02cd4a
add EMPTY_VALUE
plafer Jan 18, 2024
b009c59
fix doc
plafer Jan 18, 2024
a26a58e
fix doc
plafer Jan 18, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,6 @@ Cargo.lock

# Generated by cmake
cmake-build-*

# VS Code
.vscode/
14 changes: 11 additions & 3 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -35,19 +35,27 @@ harness = false
default = ["std"]
executable = ["dep:clap", "dep:rand_utils", "std"]
serde = ["dep:serde", "serde?/alloc", "winter_math/serde"]
std = ["blake3/std", "dep:cc", "dep:libc", "winter_crypto/std", "winter_math/std", "winter_utils/std"]
std = [
"blake3/std",
"dep:cc",
"dep:libc",
"winter_crypto/std",
"winter_math/std",
"winter_utils/std",
]

[dependencies]
blake3 = { version = "1.5", default-features = false }
clap = { version = "4.4", features = ["derive"], optional = true }
libc = { version = "0.2", default-features = false, optional = true }
libc = { version = "0.2", default-features = false, optional = true }
rand_utils = { version = "0.7", package = "winter-rand-utils", optional = true }
serde = { version = "1.0", features = [ "derive" ], default-features = false, optional = true }
serde = { version = "1.0", features = ["derive"], default-features = false, optional = true }
winter_crypto = { version = "0.7", package = "winter-crypto", default-features = false }
winter_math = { version = "0.7", package = "winter-math", default-features = false }
winter_utils = { version = "0.7", package = "winter-utils", default-features = false }

[dev-dependencies]
seq-macro = { version = "0.3" }
criterion = { version = "0.5", features = ["html_reports"] }
proptest = "1.4"
rand_utils = { version = "0.7", package = "winter-rand-utils" }
Expand Down
87 changes: 43 additions & 44 deletions benches/smt.rs
Original file line number Diff line number Diff line change
@@ -1,67 +1,66 @@
use core::mem::swap;
use criterion::{black_box, criterion_group, criterion_main, Criterion};
use miden_crypto::{merkle::SimpleSmt, Felt, Word};
use miden_crypto::{
merkle::{LeafIndex, SimpleSmt},
Felt, Word,
};
use rand_utils::prng_array;
use seq_macro::seq;

fn smt_rpo(c: &mut Criterion) {
// setup trees

let mut seed = [0u8; 32];
let mut trees = vec![];
let leaf = generate_word(&mut seed);

for depth in 14..=20 {
let leaves = ((1 << depth) - 1) as u64;
seq!(DEPTH in 14..=20 {
let leaves = ((1 << DEPTH) - 1) as u64;
for count in [1, leaves / 2, leaves] {
let entries: Vec<_> = (0..count)
.map(|i| {
let word = generate_word(&mut seed);
(i, word)
})
.collect();
let tree = SimpleSmt::with_leaves(depth, entries).unwrap();
trees.push((tree, count));
}
}

let leaf = generate_word(&mut seed);
let mut tree = SimpleSmt::<DEPTH>::with_leaves(entries).unwrap();

// benchmarks
// benchmark 1
let mut insert = c.benchmark_group("smt update_leaf".to_string());
{
let depth = DEPTH;
let key = count >> 2;
insert.bench_with_input(
format!("simple smt(depth:{depth},count:{count})"),
&(key, leaf),
|b, (key, leaf)| {
b.iter(|| {
tree.insert(black_box(LeafIndex::<DEPTH>::new(*key).unwrap()), black_box(*leaf));
});
},
);

let mut insert = c.benchmark_group("smt update_leaf".to_string());
}
insert.finish();

for (tree, count) in trees.iter_mut() {
let depth = tree.depth();
let key = *count >> 2;
insert.bench_with_input(
format!("simple smt(depth:{depth},count:{count})"),
&(key, leaf),
|b, (key, leaf)| {
b.iter(|| {
tree.update_leaf(black_box(*key), black_box(*leaf)).unwrap();
});
},
);
}
// benchmark 2
let mut path = c.benchmark_group("smt get_leaf_path".to_string());
{
let depth = DEPTH;
let key = count >> 2;
path.bench_with_input(
format!("simple smt(depth:{depth},count:{count})"),
&key,
|b, key| {
b.iter(|| {
tree.open(black_box(&LeafIndex::<DEPTH>::new(*key).unwrap()));
});
},
);

insert.finish();

let mut path = c.benchmark_group("smt get_leaf_path".to_string());

for (tree, count) in trees.iter_mut() {
let depth = tree.depth();
let key = *count >> 2;
path.bench_with_input(
format!("simple smt(depth:{depth},count:{count})"),
&key,
|b, key| {
b.iter(|| {
tree.get_leaf_path(black_box(*key)).unwrap();
});
},
);
}

path.finish();
}
path.finish();
}
});
}

criterion_group!(smt_group, smt_rpo);
Expand Down
61 changes: 32 additions & 29 deletions benches/store.rs
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
use criterion::{black_box, criterion_group, criterion_main, BatchSize, BenchmarkId, Criterion};
use miden_crypto::merkle::{DefaultMerkleStore as MerkleStore, MerkleTree, NodeIndex, SimpleSmt};
use miden_crypto::merkle::{
DefaultMerkleStore as MerkleStore, LeafIndex, MerkleTree, NodeIndex, SimpleSmt, SMT_MAX_DEPTH,
};
use miden_crypto::Word;
use miden_crypto::{hash::rpo::RpoDigest, Felt};
use rand_utils::{rand_array, rand_value};
Expand Down Expand Up @@ -28,26 +30,26 @@ fn random_index(range: u64, depth: u8) -> NodeIndex {
fn get_empty_leaf_simplesmt(c: &mut Criterion) {
let mut group = c.benchmark_group("get_empty_leaf_simplesmt");

let depth = SimpleSmt::MAX_DEPTH;
const DEPTH: u8 = SMT_MAX_DEPTH;
let size = u64::MAX;

// both SMT and the store are pre-populated with empty hashes, accessing these values is what is
// being benchmarked here, so no values are inserted into the backends
let smt = SimpleSmt::new(depth).unwrap();
let smt = SimpleSmt::<DEPTH>::new().unwrap();
let store = MerkleStore::from(&smt);
let root = smt.root();

group.bench_function(BenchmarkId::new("SimpleSmt", depth), |b| {
group.bench_function(BenchmarkId::new("SimpleSmt", DEPTH), |b| {
b.iter_batched(
|| random_index(size, depth),
|| random_index(size, DEPTH),
|index| black_box(smt.get_node(index)),
BatchSize::SmallInput,
)
});

group.bench_function(BenchmarkId::new("MerkleStore", depth), |b| {
group.bench_function(BenchmarkId::new("MerkleStore", DEPTH), |b| {
b.iter_batched(
|| random_index(size, depth),
|| random_index(size, DEPTH),
|index| black_box(store.get_node(root, index)),
BatchSize::SmallInput,
)
Expand Down Expand Up @@ -104,23 +106,22 @@ fn get_leaf_simplesmt(c: &mut Criterion) {
.enumerate()
.map(|(c, v)| (c.try_into().unwrap(), v.into()))
.collect::<Vec<(u64, Word)>>();
let smt = SimpleSmt::with_leaves(SimpleSmt::MAX_DEPTH, smt_leaves.clone()).unwrap();
let smt = SimpleSmt::<SMT_MAX_DEPTH>::with_leaves(smt_leaves.clone()).unwrap();
let store = MerkleStore::from(&smt);
let depth = smt.depth();
let root = smt.root();
let size_u64 = size as u64;

group.bench_function(BenchmarkId::new("SimpleSmt", size), |b| {
b.iter_batched(
|| random_index(size_u64, depth),
|| random_index(size_u64, SMT_MAX_DEPTH),
|index| black_box(smt.get_node(index)),
BatchSize::SmallInput,
)
});

group.bench_function(BenchmarkId::new("MerkleStore", size), |b| {
b.iter_batched(
|| random_index(size_u64, depth),
|| random_index(size_u64, SMT_MAX_DEPTH),
|index| black_box(store.get_node(root, index)),
BatchSize::SmallInput,
)
Expand All @@ -132,26 +133,26 @@ fn get_leaf_simplesmt(c: &mut Criterion) {
fn get_node_of_empty_simplesmt(c: &mut Criterion) {
let mut group = c.benchmark_group("get_node_of_empty_simplesmt");

let depth = SimpleSmt::MAX_DEPTH;
const DEPTH: u8 = SMT_MAX_DEPTH;

// both SMT and the store are pre-populated with the empty hashes, accessing the internal nodes
// of these values is what is being benchmarked here, so no values are inserted into the
// backends.
let smt = SimpleSmt::new(depth).unwrap();
let smt = SimpleSmt::<DEPTH>::new().unwrap();
let store = MerkleStore::from(&smt);
let root = smt.root();
let half_depth = depth / 2;
let half_depth = DEPTH / 2;
let half_size = 2_u64.pow(half_depth as u32);

group.bench_function(BenchmarkId::new("SimpleSmt", depth), |b| {
group.bench_function(BenchmarkId::new("SimpleSmt", DEPTH), |b| {
b.iter_batched(
|| random_index(half_size, half_depth),
|index| black_box(smt.get_node(index)),
BatchSize::SmallInput,
)
});

group.bench_function(BenchmarkId::new("MerkleStore", depth), |b| {
group.bench_function(BenchmarkId::new("MerkleStore", DEPTH), |b| {
b.iter_batched(
|| random_index(half_size, half_depth),
|index| black_box(store.get_node(root, index)),
Expand Down Expand Up @@ -212,10 +213,10 @@ fn get_node_simplesmt(c: &mut Criterion) {
.enumerate()
.map(|(c, v)| (c.try_into().unwrap(), v.into()))
.collect::<Vec<(u64, Word)>>();
let smt = SimpleSmt::with_leaves(SimpleSmt::MAX_DEPTH, smt_leaves.clone()).unwrap();
let smt = SimpleSmt::<SMT_MAX_DEPTH>::with_leaves(smt_leaves.clone()).unwrap();
let store = MerkleStore::from(&smt);
let root = smt.root();
let half_depth = smt.depth() / 2;
let half_depth = SMT_MAX_DEPTH / 2;
let half_size = 2_u64.pow(half_depth as u32);

group.bench_function(BenchmarkId::new("SimpleSmt", size), |b| {
Expand Down Expand Up @@ -286,23 +287,24 @@ fn get_leaf_path_simplesmt(c: &mut Criterion) {
.enumerate()
.map(|(c, v)| (c.try_into().unwrap(), v.into()))
.collect::<Vec<(u64, Word)>>();
let smt = SimpleSmt::with_leaves(SimpleSmt::MAX_DEPTH, smt_leaves.clone()).unwrap();
let smt = SimpleSmt::<SMT_MAX_DEPTH>::with_leaves(smt_leaves.clone()).unwrap();
let store = MerkleStore::from(&smt);
let depth = smt.depth();
let root = smt.root();
let size_u64 = size as u64;

group.bench_function(BenchmarkId::new("SimpleSmt", size), |b| {
b.iter_batched(
|| random_index(size_u64, depth),
|index| black_box(smt.get_path(index)),
|| random_index(size_u64, SMT_MAX_DEPTH),
|index| {
black_box(smt.open(&LeafIndex::<SMT_MAX_DEPTH>::new(index.value()).unwrap()))
},
BatchSize::SmallInput,
)
});

group.bench_function(BenchmarkId::new("MerkleStore", size), |b| {
b.iter_batched(
|| random_index(size_u64, depth),
|| random_index(size_u64, SMT_MAX_DEPTH),
|index| black_box(store.get_path(root, index)),
BatchSize::SmallInput,
)
Expand Down Expand Up @@ -352,7 +354,7 @@ fn new(c: &mut Criterion) {
.map(|(c, v)| (c.try_into().unwrap(), v.into()))
.collect::<Vec<(u64, Word)>>()
},
|l| black_box(SimpleSmt::with_leaves(SimpleSmt::MAX_DEPTH, l)),
|l| black_box(SimpleSmt::<SMT_MAX_DEPTH>::with_leaves(l)),
BatchSize::SmallInput,
)
});
Expand All @@ -367,7 +369,7 @@ fn new(c: &mut Criterion) {
.collect::<Vec<(u64, Word)>>()
},
|l| {
let smt = SimpleSmt::with_leaves(SimpleSmt::MAX_DEPTH, l).unwrap();
let smt = SimpleSmt::<SMT_MAX_DEPTH>::with_leaves(l).unwrap();
black_box(MerkleStore::from(&smt));
},
BatchSize::SmallInput,
Expand Down Expand Up @@ -433,24 +435,25 @@ fn update_leaf_simplesmt(c: &mut Criterion) {
.enumerate()
.map(|(c, v)| (c.try_into().unwrap(), v.into()))
.collect::<Vec<(u64, Word)>>();
let mut smt = SimpleSmt::with_leaves(SimpleSmt::MAX_DEPTH, smt_leaves.clone()).unwrap();
let mut smt = SimpleSmt::<SMT_MAX_DEPTH>::with_leaves(smt_leaves.clone()).unwrap();
let mut store = MerkleStore::from(&smt);
let depth = smt.depth();
let root = smt.root();
let size_u64 = size as u64;

group.bench_function(BenchmarkId::new("SimpleSMT", size), |b| {
b.iter_batched(
|| (rand_value::<u64>() % size_u64, random_word()),
|(index, value)| black_box(smt.update_leaf(index, value)),
|(index, value)| {
black_box(smt.insert(LeafIndex::<SMT_MAX_DEPTH>::new(index).unwrap(), value))
},
BatchSize::SmallInput,
)
});

let mut store_root = root;
group.bench_function(BenchmarkId::new("MerkleStore", size), |b| {
b.iter_batched(
|| (random_index(size_u64, depth), random_word()),
|| (random_index(size_u64, SMT_MAX_DEPTH), random_word()),
|(index, value)| {
// The MerkleTree automatically updates its internal root, the Store maintains
// the old root and adds the new one. Here we update the root to have a fair
Expand Down
Loading
Loading