Skip to content

Conversation

and-cb
Copy link
Collaborator

@and-cb and-cb commented Jul 31, 2025

This PR makes hash computations happen in parallel in a thread pool. While hashes are computed in the background, a temporary, zero-byte hash is written to the pages. The zero hashes are replaced with the actual hashes once all changes have been processed.

Status of this PR: All tests are passing, but this is a very early draft that needs a lot of cleanup and refactoring.

TODO

  • Put the DeferredChanges struct inside Context.
  • Add helper methods so that the pair slotted_page.set_value(...); deferred.set_value(...) becomes 1 call.
  • Currently DeferredRlpNode has replaced RlpNode almost everywhere. In practice, DeferredRlpNode is only needed when writing values, but when reading, RlpNode is enough, and has a lower overhead. Pointer should become a generic Pointer<R> so that both RlpNode and DeferredRlpNode can be used as appropriate.'
  • Remove unnecessary premature computation of hashes.
  • Currently, once a hash is computed, the whole node is rewritten, instead of just the hash. This can be improved.
  • Remove lazy_static.
  • Remove debug statements.
  • Make the thread pool configurable (e.g. setting the number of threads)

Base automatically changed from andrea/executor to main August 7, 2025 20:43
@cb-heimdall
Copy link
Collaborator

cb-heimdall commented Aug 7, 2025

🟡 Heimdall Review Status

Requirement Status More Info
Reviews 🟡 0/1
Denominator calculation
Show calculation
1 if user is bot 0
1 if user is external 0
2 if repo is sensitive 0
From .codeflow.yml 1
Additional review requirements
Show calculation
Max 0
0
From CODEOWNERS 0
Global minimum 0
Max 1
1
1 if commit is unverified 1
Sum 2

@and-cb and-cb force-pushed the andrea/deferred-rlp-hash-write branch from 708c187 to c22c407 Compare September 22, 2025 23:53
@and-cb and-cb changed the base branch from main to andrea/storage-engine-threadpool September 22, 2025 23:54
@and-cb and-cb force-pushed the andrea/deferred-rlp-hash-write branch 5 times, most recently from fd831a3 to cb303bf Compare September 23, 2025 01:06
@and-cb and-cb force-pushed the andrea/deferred-rlp-hash-write branch from cb303bf to cc63dcf Compare September 23, 2025 01:08
@and-cb and-cb marked this pull request as ready for review September 23, 2025 01:21
@and-cb and-cb force-pushed the andrea/storage-engine-threadpool branch from 93115b9 to 0d14745 Compare September 23, 2025 20:59
Base automatically changed from andrea/storage-engine-threadpool to main September 23, 2025 20:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants