Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make the merkle tree implementation perform better. #2273

Merged
merged 198 commits into from
Jul 23, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
198 commits
Select commit Hold shift + click to select a range
a149ebd
attempt
rauljordan Apr 16, 2024
dec2e82
Make allow_merkleize a command-line switch.
eljobe Apr 17, 2024
3c30265
Add the total time for each step-size.
eljobe Apr 18, 2024
51069f5
Add some doc comments and tests to merkle.rs
eljobe Apr 18, 2024
2ecc4f5
Allow callers to extend the Merkle Tree by adding leaves.
eljobe Apr 19, 2024
05b157f
Extend the memory merkle instead of clearing it.
eljobe Apr 19, 2024
965a202
Fix merge problems from nitro cherry-picks
eljobe Apr 21, 2024
477e49c
Remove log line about resizing memory.
eljobe Apr 21, 2024
15ad8ba
Introduce mark-and-sweep dirty branch tracking.
eljobe Apr 22, 2024
cb1d10c
Fix the implementation of extend.
eljobe Apr 22, 2024
e644b89
Add criterion benchmark for a big merkle tree.
eljobe Apr 22, 2024
d2338f1
Merge branch 'merkle-perf' into merkle-perf-a
eljobe Apr 22, 2024
b60438f
Switch to a binary tree set.
eljobe Apr 22, 2024
28c90ae
Use memory more efficiently.
eljobe Apr 23, 2024
e8868a8
Include the creation of a fresh tree for each iteration.
eljobe Apr 23, 2024
c4ab6bc
Merge branch 'merkle-perf' into merkle-perf-a
eljobe Apr 23, 2024
03d2420
Remove println
eljobe Apr 23, 2024
b7e7cd3
Further optimization.
eljobe Apr 23, 2024
b354b09
Fix the empty dirty indices bug.
eljobe Apr 23, 2024
1f52875
Add some profiling and instrumentation code.
eljobe Apr 24, 2024
4c6390e
Merge branch 'merkle-perf' into merkle-perf-a
eljobe Apr 24, 2024
976cb36
Make features work
eljobe Apr 25, 2024
aa7303b
Merge branch 'merkle-perf' into merkle-perf-a
eljobe Apr 25, 2024
d1a2f66
Add a benchmark for new_advanced.
eljobe Apr 26, 2024
b31a97b
Use capacity hints for all vector initialization.
eljobe Apr 26, 2024
e5ec9c4
Add static vectors of empty hashes for 64 layers of each MerkleType.
eljobe Apr 26, 2024
5e14a2b
Add a benchmark for new_advanced.
eljobe Apr 26, 2024
67200ca
Merge branch 'merkle-perf' into merkle-perf-a
eljobe Apr 26, 2024
c949b54
Add constant arrays of empty hashes for each Merkle type.
eljobe Apr 26, 2024
c5d2993
Fix order of VALUES and FUNCTIONS.
eljobe Apr 26, 2024
0cce10c
Make merkle.rs compile with non-rayon.
eljobe Apr 26, 2024
2f2e173
Actually set the cached merkle back on the instance.
eljobe Apr 26, 2024
c9076a3
Merge branch 'merkle-perf' into merkle-perf-a
eljobe Apr 26, 2024
fc16ec3
Update the version of the enum-iterator-derive crate.
eljobe Apr 26, 2024
368ec40
Merge branch 'merkle-perf' into merkle-perf-a
eljobe Apr 26, 2024
1a52847
Update the logic for expand to include upper layers.
eljobe Apr 29, 2024
6b627a7
Merge branch 'merkle-perf' into merkle-perf-a
eljobe Apr 29, 2024
bcf8455
Fix the merge issue for index out of bounds.
eljobe Apr 29, 2024
72bc92f
Remove the bold submodule.
eljobe Apr 30, 2024
6f7cafa
Enable always_merkelize for all uses of the merkle tree.
eljobe Apr 30, 2024
6a3c245
Merge branch 'stylus' into merkle-perf
eljobe Apr 30, 2024
36578d7
Make clippy happy.
eljobe Apr 30, 2024
97d93d5
Add the fake stuff for the benchmarks to the Dockerfile.
eljobe Apr 30, 2024
e3d7cf4
Make clippy even happier.
eljobe Apr 30, 2024
521a7d8
Cargo fmt changes only.
eljobe Apr 30, 2024
cf50743
Fix right-shifting to zero.
eljobe May 1, 2024
cc71b32
Turn off always_merkelize for now.
eljobe May 1, 2024
77866ec
Merge branch 'stylus' into merkle-perf
eljobe May 2, 2024
e197c50
Merge branch 'merkle-perf' into merkle-perf-a
eljobe May 2, 2024
d8069c8
Fix up the branch after merges.
eljobe May 2, 2024
b14bc8f
Fix the merkle_bench.
eljobe May 2, 2024
14931bc
Merge branch 'merkle-perf' into merkle-perf-a
eljobe May 2, 2024
0a31e53
Maybe 10x performance improvement.
eljobe May 2, 2024
19fcd64
Update the zerohashes to use the ones from memory.rs.
eljobe May 2, 2024
9bca51b
Cleanup outdated comment.
eljobe May 2, 2024
13de7de
Remove the extend method.
eljobe May 2, 2024
bc55504
Add a test for serialization and make it pass.
eljobe May 2, 2024
5926a83
Merge branch 'merkle-perf' into merkle-perf-a
eljobe May 3, 2024
9ebd8ba
Fix the bugged MEMORY_HASHES in the zerohashes.rs
eljobe May 3, 2024
2ce703f
Remove flat_merkle.rs.
eljobe May 3, 2024
f63141c
Merge branch 'master' into merkle-perf
eljobe May 3, 2024
e67b0e1
Remove inlined capacity function.
eljobe May 3, 2024
fc9e88f
Remove the locked_set method.
eljobe May 3, 2024
60efd57
Merge branch 'sys-test-debug' into merkle-perf
eljobe May 6, 2024
8bbf934
Merge branch 'master' into merkle-perf
eljobe May 7, 2024
35ce996
Use the json format from the go code for ValidationInputs.
eljobe May 8, 2024
b9bad0c
Fix a bug in Memory::store_slice_aligned()
eljobe May 8, 2024
c17383e
Merge branch 'merkle-perf' into merkle-perf-a
eljobe May 8, 2024
8e2042b
Merge branch 'master' into merkle-perf
eljobe May 8, 2024
ac6c0c1
Merge branch 'merkle-perf' into merkle-perf-a
eljobe May 8, 2024
bddc2c5
Merge branch 'master' into merkle-perf
eljobe May 10, 2024
0badabb
Merge branch 'merkle-perf' into merkle-perf-a
eljobe May 10, 2024
910117e
Merge branch 'master' into merkle-perf
eljobe May 10, 2024
76ee1df
Merge branch 'merkle-perf' into merkle-perf-a
eljobe May 10, 2024
9453b40
Add more counters.
eljobe May 10, 2024
e8c750a
Remove flush_module macro.
eljobe May 10, 2024
6ba0a69
Speed up block validation by another 27%.
eljobe May 13, 2024
d247ebd
Enable always_merkleize everywhere.
eljobe May 13, 2024
7013276
Make always_merkleize the only strategy.
eljobe May 13, 2024
38d1c08
Merge branch 'master' into merkle-perf
eljobe May 16, 2024
2c5e055
Merge branch 'merkle-perf' into merkle-perf-a
eljobe May 16, 2024
5bda318
Merge branch 'merkle-perf-a' into merkle-perf-b
eljobe May 16, 2024
4087fd4
Fix the Cargo.lock file
eljobe May 17, 2024
ffe7d07
Merge branch 'merkle-perf' into merkle-perf-a
eljobe May 17, 2024
febd3d8
Merge branch 'merkle-perf-a' into merkle-perf-b
eljobe May 17, 2024
ccc1ee0
Regenerate the Cargo.lock
eljobe May 17, 2024
225528e
Merge branch 'master' into merkle-perf
eljobe May 21, 2024
d8c832a
Move the layer data and dirty indices to a struct.
eljobe May 21, 2024
d4a5fd0
Merge branch 'merkle-perf' into merkle-perf-a
eljobe May 21, 2024
0f769e8
Merge branch 'merkle-perf-a' into merkle-perf-b
eljobe May 21, 2024
bd49d47
Remove deadlocks.
eljobe May 21, 2024
0a0b8e4
Merge branch 'merkle-perf' into merkle-perf-a
eljobe May 21, 2024
75ac5b6
Merge branch 'merkle-perf-a' into merkle-perf-b
eljobe May 21, 2024
d2398d8
Remove --always-merkleize
eljobe May 21, 2024
9539e9a
Make clone create completely independent layers structs.
eljobe May 22, 2024
ef7313d
Make clone create completely independent layers structs.
eljobe May 22, 2024
4306ec3
Handle the empty leaves case.
eljobe May 22, 2024
bbf8919
Merge branch 'merkle-perf' into merkle-perf-a
eljobe May 22, 2024
b0bc191
Merge branch 'master' into merkle-perf
eljobe May 22, 2024
ba38163
Merge branch 'merkle-perf' into merkle-perf-a
eljobe May 22, 2024
db24c7c
Merge branch 'master' into merkle-perf
eljobe May 24, 2024
31024b3
Merge branch 'merkle-perf' into merkle-perf-a
eljobe May 24, 2024
2efab2a
Merge branch 'master' into merkle-perf
eljobe May 27, 2024
bd45054
Merge branch 'merkle-perf' into merkle-perf-a
eljobe May 27, 2024
b075b20
Make clippy happy.
eljobe May 27, 2024
5dd0674
Merge branch 'master' into merkle-perf
eljobe May 29, 2024
4ec9470
Merge branch 'merkle-perf' into merkle-perf-a
eljobe May 29, 2024
74d4e93
Merge branch 'master' into merkle-perf
eljobe Jun 3, 2024
a2d8601
Merge branch 'merkle-perf' into merkle-perf-a
eljobe Jun 3, 2024
4e398dc
Merge branch 'master' into merkle-perf
eljobe Jun 4, 2024
21ba69b
Merge branch 'merkle-perf' into merkle-perf-a
eljobe Jun 4, 2024
db2a1df
Update Cargo.lock
eljobe Jun 4, 2024
c9c7052
Merge branch 'merkle-perf' into merkle-perf-a
eljobe Jun 4, 2024
8d074df
Fix Clone for the Memory type.
eljobe Jun 5, 2024
5e38e6e
Merge branch 'master' into merkle-perf
eljobe Jun 6, 2024
50627f2
Merge branch 'merkle-perf' into merkle-perf-a
eljobe Jun 6, 2024
1974d33
Remove race condition in merkelize.
eljobe Jun 6, 2024
fefabe7
Merge branch 'master' into merkle-perf
eljobe Jun 7, 2024
256c6aa
Merge branch 'merkle-perf' into merkle-perf-a
eljobe Jun 7, 2024
c397c8c
Merge branch 'master' into merkle-perf
eljobe Jun 10, 2024
d71cf50
Merge branch 'merkle-perf' into merkle-perf-a
eljobe Jun 10, 2024
534126f
Merge branch 'master' into merkle-perf
eljobe Jun 11, 2024
bac89a8
Merge branch 'merkle-perf' into merkle-perf-a
eljobe Jun 11, 2024
fc5dd2a
Update Cargo.lock
eljobe Jun 11, 2024
76e3bcd
Merge branch 'master' into merkle-perf
eljobe Jun 11, 2024
5946941
Merge branch 'merkle-perf' into merkle-perf-a
eljobe Jun 11, 2024
fd26cde
Merge branch 'master' into merkle-perf
eljobe Jun 11, 2024
9816df1
Merge branch 'merkle-perf' into merkle-perf-a
eljobe Jun 11, 2024
9914747
Merge branch 'master' into merkle-perf
eljobe Jun 12, 2024
81e7cd7
Merge branch 'merkle-perf' into merkle-perf-a
eljobe Jun 12, 2024
f4232df
Merge branch 'master' into merkle-perf
eljobe Jun 14, 2024
4ebae58
Merge branch 'merkle-perf' into merkle-perf-a
eljobe Jun 14, 2024
0024132
Merge branch 'master' into merkle-perf
eljobe Jun 17, 2024
f677872
Merge branch 'merkle-perf' into merkle-perf-a
eljobe Jun 17, 2024
29bc344
Merge branch 'master' into merkle-perf
eljobe Jun 18, 2024
3cc2310
Merge branch 'merkle-perf' into merkle-perf-a
eljobe Jun 18, 2024
38d0546
Merge branch 'master' into merkle-perf
eljobe Jun 19, 2024
75ef6ae
Merge branch 'merkle-perf' into merkle-perf-a
eljobe Jun 19, 2024
8471f77
Address review feedback.
eljobe Jun 19, 2024
5636424
Merge branch 'merkle-perf' into merkle-perf-a
eljobe Jun 19, 2024
b378cea
Make clippy happier.
eljobe Jun 19, 2024
ab219b2
Merge branch 'merkle-perf' into merkle-perf-a
eljobe Jun 19, 2024
7293443
Merge branch 'master' into merkle-perf
eljobe Jun 20, 2024
1e08774
Merge branch 'merkle-perf' into merkle-perf-a
eljobe Jun 20, 2024
1ed93cd
Merge branch 'master' into merkle-perf
eljobe Jun 24, 2024
46144f3
Merge branch 'merkle-perf' into merkle-perf-a
eljobe Jun 24, 2024
800f121
Merge branch 'master' into merkle-perf
eljobe Jun 27, 2024
7c7977f
Merge branch 'merkle-perf' into merkle-perf-a
eljobe Jun 27, 2024
21bf88c
Inline cloning the layers into a new Merkle.
eljobe Jun 28, 2024
49db0ff
Merge branch 'merkle-perf' into merkle-perf-a
eljobe Jun 28, 2024
a473af8
Merge branch 'master' into merkle-perf
eljobe Jun 28, 2024
68b9317
Merge branch 'merkle-perf' into merkle-perf-a
eljobe Jun 28, 2024
3afb90a
Divide the bytes of memory by the leaf size.
eljobe Jul 2, 2024
3408d25
Merge branch 'merkle-perf' into merkle-perf-a
eljobe Jul 2, 2024
25d627c
Merge branch 'master' into merkle-perf
eljobe Jul 2, 2024
c697675
Merge branch 'merkle-perf' into merkle-perf-a
eljobe Jul 2, 2024
fe8c159
Make clippy happy.
eljobe Jul 2, 2024
2acfea2
Make clippy happy.
eljobe Jul 2, 2024
41cabc4
Merge pull request #2328 from OffchainLabs/merkle-perf-a
eljobe Jul 3, 2024
56ec9f6
Merge branch 'master' into merkle-perf
eljobe Jul 8, 2024
91ddafc
Merge branch 'master' into merkle-perf
eljobe Jul 8, 2024
2c0d9de
Merge branch 'master' into merkle-perf
eljobe Jul 8, 2024
6903491
Merge branch 'master' into merkle-perf
eljobe Jul 9, 2024
4b589a8
Merge branch 'master' into merkle-perf
eljobe Jul 15, 2024
48475ea
Merge branch 'master' into merkle-perf
eljobe Jul 16, 2024
9b4f226
Merge branch 'master' into merkle-perf
eljobe Jul 16, 2024
d030924
Merge branch 'master' into merkle-perf
eljobe Jul 16, 2024
ef239b7
Merge branch 'master' into merkle-perf
eljobe Jul 18, 2024
96f5642
Remove the new_direct method from Bytes32
eljobe Jul 18, 2024
d967a82
Also remove new_direct from the zero-hash generation tests
eljobe Jul 18, 2024
dc2ed78
Remove the PartialEq implementation
eljobe Jul 18, 2024
3cd7857
Switch to the parking_lot crate's Mutex
eljobe Jul 18, 2024
9676230
Remove the unused ops:Dref
eljobe Jul 18, 2024
bd93ab7
Switch the dirty_indices to dirty_leaves
eljobe Jul 18, 2024
c58378a
Add a test to confirm that all the zero hashes are what we expect.
eljobe Jul 18, 2024
afaef97
Remove accidental import
eljobe Jul 18, 2024
cc2a84d
Organize imports better
eljobe Jul 18, 2024
420f116
Fix up the counters feature
eljobe Jul 18, 2024
a7665f3
Use a &[Bytes32] instead of a Vec
eljobe Jul 18, 2024
bb0b24b
Change how depth is calculated
eljobe Jul 18, 2024
c352007
Assert that resizing from 5 to 6 doesn't change the root hash
eljobe Jul 18, 2024
899f8c3
Remove a doulbe-lock on a Mutex
eljobe Jul 18, 2024
04b42bd
Use a more efficient calculation of capacity.
eljobe Jul 18, 2024
56f248b
Add an assertion on index being equal to length
eljobe Jul 18, 2024
981158b
Explain why comparing root hashes is the right implementation of equals
eljobe Jul 18, 2024
195a009
Switch to layers.len() - 1 during constuction
eljobe Jul 18, 2024
0a890a9
Only track the dirt for the leaves' parents on the heap
eljobe Jul 18, 2024
613e9dd
Use bitvec instead of HashMap<usize> for dirty indices
eljobe Jul 18, 2024
4ddf81f
Remove obsolete code comment
eljobe Jul 18, 2024
fe3e67c
Merge branch 'master' into merkle-perf
eljobe Jul 18, 2024
95945bd
Merge branch 'master' into merkle-perf
eljobe Jul 20, 2024
9f99749
Merge branch 'master' into merkle-perf
eljobe Jul 23, 2024
23632ca
Merge branch 'merkle-perf' of github.com:OffchainLabs/nitro into merk…
eljobe Jul 23, 2024
d63fc0e
Make two comments proper Rust doc comments
eljobe Jul 23, 2024
f3a9b4d
Simplify the resizing of the dirty_leaf_parents
eljobe Jul 23, 2024
62b5246
Just clone the dirty_leaf_parents and fill(false)
eljobe Jul 23, 2024
963a23b
Make clippy happy.
eljobe Jul 23, 2024
b04e24f
Merge branch 'master' into merkle-perf
eljobe Jul 23, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
COPY arbitrator/Cargo.* arbitrator/
COPY ./Makefile ./
COPY arbitrator/arbutil arbitrator/arbutil
COPY arbitrator/bench arbitrator/bench
COPY arbitrator/brotli arbitrator/brotli
COPY arbitrator/caller-env arbitrator/caller-env
COPY arbitrator/prover arbitrator/prover
Expand Down Expand Up @@ -131,21 +132,28 @@ RUN wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add - && \
COPY --from=brotli-library-export / target/
COPY arbitrator/Cargo.* arbitrator/
COPY arbitrator/arbutil arbitrator/arbutil
COPY arbitrator/bench arbitrator/bench
COPY arbitrator/brotli arbitrator/brotli
COPY arbitrator/caller-env arbitrator/caller-env
COPY arbitrator/prover/Cargo.toml arbitrator/prover/
COPY arbitrator/prover/benches arbitrator/prover/benches
COPY arbitrator/bench/Cargo.toml arbitrator/bench/
COPY arbitrator/jit/Cargo.toml arbitrator/jit/
COPY arbitrator/stylus/Cargo.toml arbitrator/stylus/
COPY arbitrator/tools/wasmer arbitrator/tools/wasmer
COPY arbitrator/wasm-libraries/user-host-trait/Cargo.toml arbitrator/wasm-libraries/user-host-trait/Cargo.toml
RUN bash -c 'mkdir arbitrator/{prover,jit,stylus}/src arbitrator/wasm-libraries/user-host-trait/src'
RUN echo "fn test() {}" > arbitrator/jit/src/lib.rs && \
echo "fn test() {}" > arbitrator/prover/src/lib.rs && \
echo "fn test() {}" > arbitrator/bench/src/lib.rs && \
echo "fn test() {}" > arbitrator/prover/benches/merkle_bench.rs && \
echo "fn test() {}" > arbitrator/stylus/src/lib.rs && \
echo "fn test() {}" > arbitrator/wasm-libraries/user-host-trait/src/lib.rs && \
cargo build --manifest-path arbitrator/Cargo.toml --release --lib && \
rm arbitrator/prover/src/lib.rs arbitrator/jit/src/lib.rs arbitrator/stylus/src/lib.rs && \
rm arbitrator/wasm-libraries/user-host-trait/src/lib.rs
rm arbitrator/wasm-libraries/user-host-trait/src/lib.rs && \
rm arbitrator/prover/benches/merkle_bench.rs && \
rm arbitrator/bench/src/lib.rs
COPY ./Makefile ./
COPY arbitrator/prover arbitrator/prover
COPY arbitrator/wasm-libraries arbitrator/wasm-libraries
Expand Down
14 changes: 7 additions & 7 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -428,10 +428,10 @@ $(stylus_test_erc20_wasm): $(stylus_test_erc20_src)
@touch -c $@ # cargo might decide to not rebuild the binary

contracts/test/prover/proofs/float%.json: $(arbitrator_cases)/float%.wasm $(prover_bin) $(output_latest)/soft-float.wasm
$(prover_bin) $< -l $(output_latest)/soft-float.wasm -o $@ -b --allow-hostapi --require-success --always-merkleize
$(prover_bin) $< -l $(output_latest)/soft-float.wasm -o $@ -b --allow-hostapi --require-success

contracts/test/prover/proofs/no-stack-pollution.json: $(arbitrator_cases)/no-stack-pollution.wasm $(prover_bin)
$(prover_bin) $< -o $@ --allow-hostapi --require-success --always-merkleize
$(prover_bin) $< -o $@ --allow-hostapi --require-success

target/testdata/preimages.bin:
mkdir -p `dirname $@`
Expand All @@ -455,19 +455,19 @@ contracts/test/prover/proofs/global-state.json:
echo "[]" > $@

contracts/test/prover/proofs/forward-test.json: $(arbitrator_cases)/forward-test.wasm $(arbitrator_tests_forward_deps) $(prover_bin)
$(prover_bin) $< -o $@ --allow-hostapi --always-merkleize $(patsubst %,-l %, $(arbitrator_tests_forward_deps))
$(prover_bin) $< -o $@ --allow-hostapi $(patsubst %,-l %, $(arbitrator_tests_forward_deps))

contracts/test/prover/proofs/link.json: $(arbitrator_cases)/link.wasm $(arbitrator_tests_link_deps) $(prover_bin)
$(prover_bin) $< -o $@ --allow-hostapi --always-merkleize --stylus-modules $(arbitrator_tests_link_deps) --require-success
$(prover_bin) $< -o $@ --allow-hostapi --stylus-modules $(arbitrator_tests_link_deps) --require-success

contracts/test/prover/proofs/dynamic.json: $(patsubst %,$(arbitrator_cases)/%.wasm, dynamic user) $(prover_bin)
$(prover_bin) $< -o $@ --allow-hostapi --always-merkleize --stylus-modules $(arbitrator_cases)/user.wasm --require-success
$(prover_bin) $< -o $@ --allow-hostapi --stylus-modules $(arbitrator_cases)/user.wasm --require-success

contracts/test/prover/proofs/bulk-memory.json: $(patsubst %,$(arbitrator_cases)/%.wasm, bulk-memory) $(prover_bin)
$(prover_bin) $< -o $@ --allow-hostapi --always-merkleize --stylus-modules $(arbitrator_cases)/user.wasm -b
$(prover_bin) $< -o $@ --allow-hostapi --stylus-modules $(arbitrator_cases)/user.wasm -b

contracts/test/prover/proofs/%.json: $(arbitrator_cases)/%.wasm $(prover_bin)
$(prover_bin) $< -o $@ --allow-hostapi --always-merkleize
$(prover_bin) $< -o $@ --allow-hostapi

# strategic rules to minimize dependency building

Expand Down
Loading
Loading