Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

archive: Refactor archive_storage method into subscription #6483

Merged
merged 81 commits into from
Nov 29, 2024
Merged

archive: Refactor archive_storage method into subscription #6483

merged 81 commits into from
Nov 29, 2024

Conversation

lexnv
Copy link
Contributor

@lexnv lexnv commented Nov 14, 2024

This PR adapts the archive_storage implementation from a method to a subscription.

This keeps the archive APIs uniform and consistent.

Builds on: #5997

cc @paritytech/subxt-team

@lexnv
archive/api: Add stroage diff method
e17a555 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
storage/events: Add storage events and test serialization
4101ee6 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
archive: Add initial implementation for storage diff
d781d67 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
archive: Handle modified and delete cases
60e1a93 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
archive/events: Derive more impl for diff items
63da031 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
archive: Deduplicate input keys
25b9e5a 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
archive: Move code to different module
52cfaf7 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
archive: Fetch value / hash or both
ee6bed7 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
events: Add subscription events for ArchiveStorageDiffEvent
8948e74 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
archive: Make archive_storageDiff a subscription
fbfb0dc 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
archive: Modify storageDiff to leverage subscription backpressure
f7b33ed 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
archive: Deduplicate items to a separate function
f47db36 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
archive/tests: Check deduplication
97546a4 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
archive: Fix deduplication shortest key
baa3dd0 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
archive/tests: Check complex deduplication
4683f6b 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
archive: Simplify logic of trie iteration
36f59d2 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
archive: Improve documentation
a084f47 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
archive/events: Add derive(Eq) to archive storage diff events
77cac50 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
archive/tests: Check query for main trie under prefix
e2324a4 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
archive: Add trace and debug logs for storage diff
4b05913 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
archive: Send StorageDiffDone for partial trie queries
83cb9b4 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
archive/tests: Check no changes between blocks
85c810e 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
archive/tests: Ensure added key
152d13b 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
archive/tests: Extend test with interleaved values and hashes
4f549bf 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
rpc-v2: Use indexmap crate
d208068 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
archive: Move prevHash as last parameter to archive diff
369a4d2 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
archive: Preserve order of elements with indexMap
a02cc84 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
archive/tests: Check deleted keys
637b446 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
events: Add common wrappers for construction
76efb18 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
archive: Propagate errors via events
ca1fa20 
Signed-off-by: Alexandru Vasile <[email protected]>
lexnv and others added 8 commits November 27, 2024 17:22
@lexnv
archive: Remove commented code and add documentation
c6e40a0 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv @jsdw
Update substrate/client/rpc-spec-v2/src/archive/api.rs
a07b5c4 
Co-authored-by: James Wilson <[email protected]>
@lexnv
archive: Remove unneeded lifetime
ae00327 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
Merge remote-tracking branch 'origin/lexnv/storage-diff' into lexnv/s…
fd48434 
…torage-diff
@lexnv
Merge remote-tracking branch 'origin/lexnv/storage-diff' into lexnv/s…
736a5ad 
…torage-sub

Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
archive: Check sink closed
791831f 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
archive: Refactor process methods
3ddaf2d 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
archive/tests: Adjust to new assumptions wrt StorageDone
774b525 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv lexnv marked this pull request as ready for review November 27, 2024 16:33
niklasad1
niklasad1 reviewed Nov 27, 2024
View reviewed changes
substrate/client/rpc-spec-v2/src/archive/archive.rs
log::debug!(target: LOG_TARGET, "Finished processing partial trie query");
} else if event.is_err() {
log::debug!(target: LOG_TARGET, "Error encountered while processing partial trie query");
/// Sends all the events of the storage_diff method to the sink.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A little annoying to have two different functions for ArchiveStorageDiffEvent and for ArchiveStorageEvent but I have no better suggestion.

It's fine

niklasad1
niklasad1 approved these changes Nov 27, 2024
View reviewed changes
Copy link
Member

@niklasad1 niklasad1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, nice to get rid of the static limits :)

Base automatically changed from lexnv/storage-diff to master November 27, 2024 19:04
@paritytech-workflow-stopper
Copy link

All GitHub workflows were cancelled due to failure one of the required jobs.
Failed workflow url: https://github.com/paritytech/polkadot-sdk/actions/runs/12067280021
Failed job name: test-linux-stable

@lexnv
archive/tests: Fix wrong merge
ff09c7e 
Signed-off-by: Alexandru Vasile <[email protected]>
@lexnv
archive/events: Align with spec
087f0e7 
Signed-off-by: Alexandru Vasile <[email protected]>
jsdw
jsdw approved these changes Nov 29, 2024
View reviewed changes
Copy link
Contributor

@jsdw jsdw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@lexnv lexnv added this pull request to the merge queue Nov 29, 2024
Merged via the queue into master with commit 4e7c968 Nov 29, 2024
194 of 199 checks passed
@lexnv lexnv deleted the lexnv/storage-sub branch November 29, 2024 15:41
lexnv added a commit that referenced this pull request Nov 29, 2024
@lexnv @jsdw
archive: Refactor archive_storage method into subscription (#6483)
53b0729 
This PR adapts the `archive_storage` implementation from a method to a
subscription.

This keeps the archive APIs uniform and consistent.

Builds on: #5997

cc @paritytech/subxt-team

---------

Signed-off-by: Alexandru Vasile <[email protected]>
Co-authored-by: James Wilson <[email protected]>
@lexnv lexnv mentioned this pull request Nov 29, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@niklasad1 niklasad1 niklasad1 approved these changes

@jsdw jsdw jsdw approved these changes

Assignees

@lexnv lexnv

Labels
I4-refactor Code needs refactoring. I5-enhancement An additional feature request. R0-silent Changes should not be mentioned in any release notes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@lexnv @jsdw @niklasad1