[reshardingV3] State ShardUIdMapping #12084

staffik · 2024-09-12T11:04:55Z

Tracking issue: #12050

I would like to collect an early feedback on the first steps to implementing mapping strategy for State in ReshardingV3.
Went through all references in code to DbCol::State, excluding tests for now.

Update: in-memory mapping

Digging more into the code, it turns out we construct TrieStorage very often (e.g. every time we apply a chunk):

nearcore/core/store/src/trie/shard_tries.rs

Line 125 in 3d0fd26

Arc::new(TrieDBStorage::new(self.0.store.clone(), shard_uid))

It can be initialized with a shared reference to TrieCache, that it is kept in ShardTries and protected by mutex:

nearcore/core/store/src/trie/shard_tries.rs

Line 235 in 3d0fd26

    
           TrieCachingStorage::new(self.0.store.clone(), cache, shard_uid, is_view, prefetch_api)

I used similar approach by creating StateReader that is kept in ShardTries and used to create each new instance of TrieStorage.

StateReader needs to know resharding tree so that it knows parent shard uid when it cannot find a node in the database.
It keeps the resharding tree as hashmap. Alternatively, it could store ancestors list for each shard, but it would use O(S^2) memory, which is also fine for the near future.
The problem is that the resharding history might not be easily accessible, see #12084 (comment).

What to do?

One idea is to add a db column that stores resharding parent shard_uid and update it on resharding event.
It could be initially empty, since we do not need resharding history from before reshardingV3 lands.
There are many places in code where we have only access to store (not epoch manager) and we need the resharding history, so I think this idea is the way to go.
Also, it would simplify the code as StateReader could be initially empty and would not need to know shard_uids and resharding history upfront.

Old version: keeping the mapping in database:

Summary:

Added persistently stored DbCol::ShardUIdMapping.

Used MappedShardUId wrapper in places where we expect the mapped value (where shard UID is used as a db key prefix for the State column).

Added map_shard_uid to read the mapping from db. For now, it never fails and it falls back to the input shard UID.

Next steps:

Handle updating the mapping on a resharding or state sync event.
State clean up (e.g. gc parent state when it is no longer referenced by any child).
Tests.

core/store/src/flat/store_helper.rs

core/store/src/cold_storage.rs

staffik · 2024-09-12T11:18:15Z

core/store/src/trie/mem/parallel_loader.rs

-        let (start, end) = subtree_to_load.to_iter_range(self.shard_uid);
+        let (start, end) = subtree_to_load.to_iter_range(self.shard_uid_db_prefix.0);

        // Load all the keys in this range from the FlatState column.


Not sure about that. Maybe it would read entire parent shard while we only want child shard.
And it reads FlatState so we might want to synchronize on changes here.

Yeah, agreed, this seems fragile.

Taking a step back what is our plan for loading memtrie post resharding? Perhaps we can rely on that and panic here if shard_uid != mapped_shard_uid.

staffik · 2024-09-12T11:21:05Z

There is

scan_db_column(
    col: &str,
    lower_bound: Option<&[u8]>,
    upper_bound: Option<&[u8]>,
    store: Store)

and it might not be possible to scan child shard only.
It is only a debug tool and we probably would need to live with that.

codecov · 2024-09-12T11:40:55Z

Codecov Report

Attention: Patch coverage is 80.22388% with 53 lines in your changes missing coverage. Please review.

Project coverage is 71.54%. Comparing base (3d0fd26) to head (8a4ffea).

Files with missing lines	Patch %	Lines
core/store/src/trie/state_reader.rs	83.11%	10 Missing and 3 partials ⚠️
nearcore/src/entity_debug.rs	0.00%	9 Missing ⚠️
tools/state-viewer/src/commands.rs	0.00%	7 Missing ⚠️
core/store/src/trie/from_flat.rs	0.00%	5 Missing ⚠️
tools/database/src/analyze_contract_sizes.rs	0.00%	5 Missing ⚠️
tools/database/src/state_perf.rs	0.00%	5 Missing ⚠️
tools/database/src/analyze_delayed_receipt.rs	0.00%	2 Missing ⚠️
tools/fork-network/src/cli.rs	0.00%	2 Missing ⚠️
tools/state-viewer/src/trie_iteration_benchmark.rs	0.00%	2 Missing ⚠️
core/store/src/flat/store_helper.rs	66.66%	1 Missing ⚠️
... and 2 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #12084      +/-   ##
==========================================
+ Coverage   71.51%   71.54%   +0.02%     
==========================================
  Files         818      819       +1     
  Lines      164494   164637     +143     
  Branches   164494   164637     +143     
==========================================
+ Hits       117644   117782     +138     
+ Misses      41713    41712       -1     
- Partials     5137     5143       +6

Flag	Coverage Δ
backward-compatibility	`0.17% <0.00%> (-0.01%)`	⬇️
db-migration	`0.17% <0.00%> (-0.01%)`	⬇️
genesis-check	`1.26% <0.00%> (-0.01%)`	⬇️
integration-tests	`38.68% <54.85%> (+0.02%)`	⬆️
linux	`71.33% <80.22%> (+0.01%)`	⬆️
linux-nightly	`71.11% <80.22%> (+0.01%)`	⬆️
macos	`54.14% <74.51%> (+0.27%)`	⬆️
pytests	`1.52% <0.00%> (-0.01%)`	⬇️
sanity-checks	`1.33% <0.00%> (-0.01%)`	⬇️
unittests	`65.26% <76.11%> (+0.01%)`	⬆️
upgradability	`0.21% <0.00%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

core/primitives/src/shard_layout.rs

core/store/src/trie/shard_tries.rs

chain/chain/src/garbage_collection.rs

wacban

Looks good, I left a few comments!

core/store/src/cold_storage.rs

core/store/src/flat/store_helper.rs

wacban · 2024-09-17T08:59:49Z

core/store/src/trie/mem/parallel_loader.rs

-        let (start, end) = subtree_to_load.to_iter_range(self.shard_uid);
+        let (start, end) = subtree_to_load.to_iter_range(self.shard_uid_db_prefix.0);

        // Load all the keys in this range from the FlatState column.


Yeah, agreed, this seems fragile.

Taking a step back what is our plan for loading memtrie post resharding? Perhaps we can rely on that and panic here if shard_uid != mapped_shard_uid.

core/store/src/trie/shard_tries.rs

wacban · 2024-09-17T09:09:33Z

core/store/src/trie/shard_tries.rs

+            let shard_uid_db_prefix = match self.0.shard_uid_to_db_prefix.get(&shard_uid) {
+                Some(mapped_shard_uid) => *mapped_shard_uid,
+                // TODO(reshardingV3) Think about how None should be handled here.
+                None => shard_uid.into(),
+            };


This is on a bit higher level than I anticipated. I thought the mapping would happen closer to the db itself, perhaps in Store or Database. I'm not saying this is wrong but I'm curious on your thoughts about how those two approaches compare.

core/store/src/trie/trie_storage.rs

wacban · 2024-09-17T09:12:48Z

core/store/src/trie/trie_storage.rs

+        // TODO(reshardingV3) Think about how to handle it the best way
+        _ => shard_uid,


+1 best to think about what are the invariants and handle it accordingly.

If the invariant is that the mapping should always be populated for all shards then returning an error seems quite reasonable.

core/store/src/trie/trie_storage.rs

staffik · 2024-09-19T09:27:50Z

chain/chain/src/runtime/mod.rs

@@ -101,10 +102,14 @@ impl NightshadeRuntime {
        let trie_viewer = TrieViewer::new(trie_viewer_state_size_limit, max_gas_burnt_view);
        let flat_storage_manager = FlatStorageManager::new(store.clone());
        let shard_uids: Vec<_> = genesis_config.shard_layout.shard_uids().collect();
+        // TODO(reshardingV3) Recursively calculate resharding parents for `shard_uids`.


That would require iterating through previous epochs, which is a bit cumbersome, see

nearcore/chain/client/src/sync/epoch.rs

Lines 232 to 234 in 3d0fd26

// and (2) it is not easy to walk backwards from the last epoch; there's no

// "give me the previous epoch" query. So instead, we use block header's

// `next_epoch_id` to establish an epoch chain.

.

staffik added the A-resharding Area: State resharding label Sep 12, 2024

staffik changed the title ~~[reshardingV3]~~ [reshardingV3] State ShardUIdMapping Sep 12, 2024

staffik commented Sep 12, 2024

View reviewed changes

core/store/src/flat/store_helper.rs Show resolved Hide resolved

staffik commented Sep 12, 2024

View reviewed changes

core/store/src/cold_storage.rs Outdated Show resolved Hide resolved

staffik commented Sep 12, 2024

View reviewed changes

shreyan-gupta reviewed Sep 14, 2024

View reviewed changes

core/primitives/src/shard_layout.rs Outdated Show resolved Hide resolved

tayfunelmas reviewed Sep 16, 2024

View reviewed changes

core/store/src/trie/shard_tries.rs Outdated Show resolved Hide resolved

tayfunelmas reviewed Sep 16, 2024

View reviewed changes

chain/chain/src/garbage_collection.rs Outdated Show resolved Hide resolved

walnut-the-cat mentioned this pull request Sep 16, 2024

🔷 [Project Tracking] Resharding v3 #11881

Open

65 tasks

wacban reviewed Sep 17, 2024

View reviewed changes

Mapped shard uid

5f8fabf

staffik force-pushed the reshardingv3-state branch from 7d1841c to 85574b2 Compare September 19, 2024 09:23

staffik commented Sep 19, 2024

View reviewed changes

In-memory shard UId mapping

8a4ffea

staffik force-pushed the reshardingv3-state branch from 85574b2 to 8a4ffea Compare September 19, 2024 09:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[reshardingV3] State ShardUIdMapping #12084

[reshardingV3] State ShardUIdMapping #12084

staffik commented Sep 12, 2024 •

edited

Loading

staffik Sep 12, 2024

wacban Sep 17, 2024

staffik commented Sep 12, 2024

codecov bot commented Sep 12, 2024 •

edited

Loading

wacban left a comment

wacban Sep 17, 2024

wacban Sep 17, 2024

wacban Sep 17, 2024

staffik Sep 19, 2024 •

edited

Loading

		// TODO(reshardingV3) Think about how to handle it the best way
		_ => shard_uid,

	// and (2) it is not easy to walk backwards from the last epoch; there's no
	// "give me the previous epoch" query. So instead, we use block header's
	// `next_epoch_id` to establish an epoch chain.

[reshardingV3] State ShardUIdMapping #12084

Are you sure you want to change the base?

[reshardingV3] State ShardUIdMapping #12084

Conversation

staffik commented Sep 12, 2024 • edited Loading

Update: in-memory mapping

What to do?

Old version: keeping the mapping in database:

Next steps:

staffik Sep 12, 2024

Choose a reason for hiding this comment

wacban Sep 17, 2024

Choose a reason for hiding this comment

staffik commented Sep 12, 2024

codecov bot commented Sep 12, 2024 • edited Loading

Codecov Report

wacban left a comment

Choose a reason for hiding this comment

wacban Sep 17, 2024

Choose a reason for hiding this comment

wacban Sep 17, 2024

Choose a reason for hiding this comment

wacban Sep 17, 2024

Choose a reason for hiding this comment

staffik Sep 19, 2024 • edited Loading

Choose a reason for hiding this comment

staffik commented Sep 12, 2024 •

edited

Loading

codecov bot commented Sep 12, 2024 •

edited

Loading

staffik Sep 19, 2024 •

edited

Loading