accounts-db: relax intrabatch account locks #4253

2501babe · 2025-01-03T10:44:03Z

Problem

simd83 proposes removing the constraint that transactions in the same entry cannot have read/write or write/write contention on any account. a previous pr modified svm to be able to carry over state changes between such transactions while processing a transaction batch. this pr modifies consensus to remove the account locking rules that prevent such batches from being created or replayed

Summary of Changes

add a new function to AccountLocks, try_lock_transaction_batch, which only checks for locking conflicts with other batches, allowing any account overlap within the batch itself. modify Accounts and Bank to use it instead when the feature gate relax_intrabatch_account_locks is activated. also modify prepare_sanitized_batch_with_results to deduplicate transactions within a batch by signature to prevent replay attacks, such that two instances of the same transaction cause the first to lock out the second, in a similar manner to the non-simd83 behavior for this special case

since transaction results are used more extensively than previous, some functions with *_wiith_results variants have bene collapsed into wrappers around that variant. we also refactor several things to favor iterators over vectors, to avoid places where iters are collected and transformed back into iters

important code changes are confined to accounts-db/src/accounts.rs, accounts-db/src/account_locks.rs, and runtime/src/bank.rs. changes in core, ledger, and runtime transaction batch only affect tests. overall the large majority of changes are fixes or improvements to tests

Feature Gate Issue: https://github.com/anza-xyz/feature-gate-tracker/issues/76

accounts-db/src/accounts.rs

apfitzge · 2025-01-08T15:54:15Z

accounts-db/src/account_locks.rs

+        // HANA TODO the vec allocation here is unfortunate but hard to avoid
+        // we cannot do this in one closure because of borrow rules
+        // play around with alternate strategies, according to benches this may be up to
+        // 50% slower for small batches and few locks, but for large batches and many locks


bench using jemalloc? i'd think it would do a reasonable job of just keeping the mem in thread-local cache for re-use

2501babe · 2025-01-23T11:42:21Z

accounts-db/src/account_locks.rs

 #[derive(Debug, Default)]
 pub struct AccountLocks {
-    write_locks: AHashSet<Pubkey>,
+    write_locks: AHashMap<Pubkey, u64>,
    readonly_locks: AHashMap<Pubkey, u64>,
 }


because the read- and write-lock hashmaps have the same type now, all the functions that change them are basically the same. we could use an enum or hashmap reference to discriminate and delete half of the functions, but i left it like this for your review before butchering it

2501babe · 2025-01-23T12:10:28Z

accounts-db/src/accounts.rs

+        relax_intrabatch_account_locks: bool,
    ) -> Vec<Result<()>> {
        // Validate the account locks, then get iterator if successful validation.
-        let tx_account_locks_results: Vec<Result<_>> = txs


Accounts::lock_accounts() could be reimpled as a wrapper on lock_accounts_with_results() or possibly deleted, it isnt really required anymore since all batch-building needs to have results. but we could leave it as-is, or could do some kind of refactor with TransactionAccountLocksIterator

2501babe · 2025-01-23T12:13:02Z

accounts-db/src/accounts.rs

-    fn lock_accounts_inner(
+    fn lock_accounts_inner<'a>(
        &self,
-        tx_account_locks_results: Vec<Result<TransactionAccountLocksIterator<impl SVMMessage>>>,
+        tx_account_locks_results: impl Iterator<
+            Item = Result<TransactionAccountLocksIterator<'a, impl SVMMessage + 'a>>,
+        >,
+        relax_intrabatch_account_locks: bool,


in general where possible i changed vecs to iters, we eliminate several uses of collect() and just pass around the closure chains. this makes the type signatures look kind of stupid tho and there are possibly things we could refactor to be better (like maybe combining transactions with the transaction results instead of taking for granted they always have the same length). im undecided about style tho

2501babe · 2025-01-23T12:14:19Z

accounts-db/src/accounts.rs

+        if relax_intrabatch_account_locks {
+            let validated_batch_keys = tx_account_locks_results.map(|tx_account_locks_result| {
+                tx_account_locks_result
+                    .map(|tx_account_locks| tx_account_locks.accounts_with_is_writable())
+            });
+
+            account_locks.try_lock_transaction_batch(validated_batch_keys)
+        } else {
+            tx_account_locks_results
+                .map(|tx_account_locks_result| match tx_account_locks_result {
+                    Ok(tx_account_locks) => account_locks
+                        .try_lock_accounts(tx_account_locks.accounts_with_is_writable()),
+                    Err(err) => Err(err),
+                })
+                .collect()
+        }


this is the main branch that the feature controls. we pass it in as a param so Accounts doesnt need FeatureSet, only Bank. the only other place we use the feature is to enable signature-based transaction deduplication

2501babe · 2025-01-23T12:19:55Z

runtime/src/bank.rs

+        // with simd83 enabled, we must deduplicate transactions by signature
+        // previously, conflicting account locks would do it as a side effect
+        let mut batch_signatures = AHashSet::with_capacity(transactions.len());
+        let transaction_results =
+            transaction_results
+                .enumerate()
+                .map(|(i, tx_result)| match tx_result {
+                    Ok(())
+                        if relax_intrabatch_account_locks
+                            && !batch_signatures.insert(transactions[i].signature()) =>
+                    {
+                        Err(TransactionError::AccountInUse)
+                    }
+                    Ok(()) => Ok(()),
+                    Err(e) => Err(e),
+                });


this is the dedupe step mentioned in a comment above. we could use a double for loop instead of a hashset but this seemed much more straightforward since the inner loop would have to abort based on the outer loop index

I'd guess it's almost certainly faster to just do a brute-force since our batches are small (replay will be size 1 for unified-scheduler); but might get complicated with the current iterator interface. Fine to leave it as is.

2501babe · 2025-01-24T09:31:34Z

recent sample of bench comparisons, master vs this branch with simd83 enabled

bench_lock_accounts/batch_size_1_locks_count_2
                        time:   [188.03 µs 188.11 µs 188.19 µs]
                        thrpt:  [5.4414 Melem/s 5.4438 Melem/s 5.4461 Melem/s]
bench_lock_accounts/batch_size_1_locks_count_2_old
                        time:   [194.10 µs 194.20 µs 194.32 µs]
                        thrpt:  [5.2696 Melem/s 5.2729 Melem/s 5.2757 Melem/s]

bench_lock_accounts/batch_size_32_locks_count_64
                        time:   [5.3985 ms 5.6560 ms 5.9096 ms]
                        thrpt:  [173.28 Kelem/s 181.05 Kelem/s 189.68 Kelem/s]
bench_lock_accounts/batch_size_32_locks_count_64_simd83
                        time:   [3.0115 ms 3.0123 ms 3.0132 ms]
                        thrpt:  [339.84 Kelem/s 339.94 Kelem/s 340.03 Kelem/s]

bench_lock_accounts/batch_size_64_locks_count_64_read_conflicts
                        time:   [2.8057 ms 2.8066 ms 2.8075 ms]
                        thrpt:  [364.74 Kelem/s 364.86 Kelem/s 364.97 Kelem/s]
bench_lock_accounts/batch_size_64_locks_count_64_read_conflicts_simd83
                        time:   [2.8175 ms 2.8181 ms 2.8188 ms]
                        thrpt:  [363.28 Kelem/s 363.36 Kelem/s 363.44 Kelem/s]

in general we perform slightly worse for tiny batches and as well or better for large batches. note these benches call code in Accounts and AccountLocks but not Bank

accounts-db/src/account_locks.rs

apfitzge · 2025-01-30T17:45:31Z

runtime/src/bank.rs

+        // with simd83 enabled, we must deduplicate transactions by signature
+        // previously, conflicting account locks would do it as a side effect


we must dedup because we check for already_processed in a batch, then process, then add to the status_cache.

Is that correct summary of why we need to do this now?

im not fully confident in my understanding of the status cache but i believe the way it works is you execute the batch and the signatures of processed (non-dropped) transactions go in the status cache only after theyve all been run. there are no checks in svm (nor should there be) for duplicate transactions within a batch

if replay is going to single-batch everything i guess it would enforce this as a side effect but it seemed good to do it here, since this code already did enforce this constraint (a malicious block that put the same transaction in one entry multiple times would fail locking in replay, without anything involving status cache, because the transactions would take the same write lock on the fee-payer)

Yeah we should enforce it for sure

apfitzge · 2025-01-30T17:47:12Z

runtime/src/bank.rs

+
+        // with simd83 enabled, we must deduplicate transactions by signature
+        // previously, conflicting account locks would do it as a side effect
+        let mut batch_signatures = AHashSet::with_capacity(transactions.len());


let's use message_hash here instead of signature.

status-cache uses both, but signature is only necessary for RPC operation for fast signature lookup.
iirc reason to use message hash is because of signature malleability.

i used it because message_hash isnt provided by SVMMessage or SVMTransaction. would you like me to add it to SVMTransaction? its available on SanitizedTransaction but providing it from SanitizedMessage would require us to add it to LegacyMessage and v0::LoadedMessage

We can probably just change the trait bound to TransactionWithMeta on this function, that trait should provide it, and I'm fairly certain the things we actually call this with impl it.

had to squash the past because rebasing was getting ugly but this is bae70c0

We have the function directly as part of TransactionWithMeta, just call message_hash() on the TransactionWithMeta tx.

as_sanitized_transaction (unless it IS a sanitized transaction) will create a sanitized transaction. and possibly do 100s of allocations.
That fn is only there because of legacy interfaces - we shouldn't use it anywhere it's not strictly necessary (should be only geyser rn)

gotcha, i see it now. i only looked at TransactionWithMeta rather than StaticMeta

There's a comment on that fn. I'll update it to be more clear that basically no one should be using that, except for the couple places we call into geyser.

Made comments on that trait better here - #4827

joncinque · 2025-02-04T20:49:32Z

This PR contains changes to the solana sdk, which will be moved to a new repo within the next week, when v2.2 is branched from master.

Please merge or close this PR as soon as possible, or re-create the sdk changes when the new repository is ready at https://github.com/anza-xyz/solana-sdk

2501babe · 2025-02-05T15:07:20Z

@joncinque this is a core runtime pr, the only sdk change is adding the feature gate. i assume when the new repo is created the procedure is going to be like:

pr the new feature gate to sdk which will be accepted and merged without needing to see any outside code
keep this pr, rebase on new sdk dependency and remove changes to sdk/feature-set/ which will no longer exist
continue with code review as normal

right?

joncinque · 2025-02-05T15:31:03Z

Yep that sounds right. I'm also adding a new FeatureSet constructor so that agave can define its own features in this repo and avoid that annoying back and forth. 90% of PRs with changes to the sdk were just feature additions.

2501babe force-pushed the 20250103_simd83locking branch from 3e016b3 to f748a5f Compare January 3, 2025 10:46

2501babe self-assigned this Jan 3, 2025

2501babe force-pushed the 20250103_simd83locking branch 4 times, most recently from 2fb8527 to 11d1dcb Compare January 7, 2025 11:23

2501babe changed the title ~~accounts-db: only lock accounts across threads~~ accounts-db: disable intrabatch account locks Jan 7, 2025

talalim reviewed Jan 8, 2025

View reviewed changes

accounts-db/src/accounts.rs Outdated Show resolved Hide resolved

apfitzge reviewed Jan 8, 2025

View reviewed changes

2501babe force-pushed the 20250103_simd83locking branch from 95b17b3 to d1ec289 Compare January 13, 2025 12:13

2501babe changed the title ~~accounts-db: disable intrabatch account locks~~ accounts-db: relax intrabatch account locks Jan 13, 2025

2501babe force-pushed the 20250103_simd83locking branch 2 times, most recently from a11ef4d to c234cc6 Compare January 17, 2025 14:23

2501babe commented Jan 23, 2025

View reviewed changes

2501babe added the feature-gate Pull Request adds or modifies a runtime feature gate label Jan 23, 2025

2501babe force-pushed the 20250103_simd83locking branch 2 times, most recently from d378d3a to c6d7105 Compare January 24, 2025 09:04

2501babe marked this pull request as ready for review January 24, 2025 12:19

2501babe requested a review from apfitzge January 24, 2025 12:19

apfitzge reviewed Jan 30, 2025

View reviewed changes

accounts-db/src/account_locks.rs Show resolved Hide resolved

apfitzge reviewed Jan 30, 2025

View reviewed changes

2501babe added 2 commits February 5, 2025 05:05

impl simd83 locking

2ea4f77

change signature to message hash

bae70c0

2501babe force-pushed the 20250103_simd83locking branch from d2cf375 to bae70c0 Compare February 5, 2025 15:03

use StaticMeta instead

d33a309

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

accounts-db: relax intrabatch account locks #4253

accounts-db: relax intrabatch account locks #4253

2501babe commented Jan 3, 2025 •

edited

Loading

apfitzge Jan 8, 2025

2501babe Jan 23, 2025 •

edited

Loading

2501babe Jan 23, 2025 •

edited

Loading

2501babe Jan 23, 2025

2501babe Jan 23, 2025

2501babe Jan 23, 2025

apfitzge Jan 30, 2025

2501babe commented Jan 24, 2025 •

edited

Loading

apfitzge Jan 30, 2025

2501babe Jan 30, 2025 •

edited

Loading

apfitzge Jan 30, 2025

apfitzge Jan 30, 2025

2501babe Jan 30, 2025 •

edited

Loading

apfitzge Jan 31, 2025

2501babe Feb 5, 2025

apfitzge Feb 5, 2025

2501babe Feb 6, 2025

apfitzge Feb 6, 2025

apfitzge Feb 6, 2025

joncinque commented Feb 4, 2025

2501babe commented Feb 5, 2025 •

edited

Loading

joncinque commented Feb 5, 2025

		// with simd83 enabled, we must deduplicate transactions by signature
		// previously, conflicting account locks would do it as a side effect

accounts-db: relax intrabatch account locks #4253

Are you sure you want to change the base?

accounts-db: relax intrabatch account locks #4253

Conversation

2501babe commented Jan 3, 2025 • edited Loading

Problem

Summary of Changes

Choose a reason for hiding this comment

2501babe Jan 23, 2025 • edited Loading

Choose a reason for hiding this comment

2501babe Jan 23, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

2501babe commented Jan 24, 2025 • edited Loading

Choose a reason for hiding this comment

2501babe Jan 30, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

2501babe Jan 30, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joncinque commented Feb 4, 2025

2501babe commented Feb 5, 2025 • edited Loading

joncinque commented Feb 5, 2025

2501babe commented Jan 3, 2025 •

edited

Loading

2501babe Jan 23, 2025 •

edited

Loading

2501babe Jan 23, 2025 •

edited

Loading

2501babe commented Jan 24, 2025 •

edited

Loading

2501babe Jan 30, 2025 •

edited

Loading

2501babe Jan 30, 2025 •

edited

Loading

2501babe commented Feb 5, 2025 •

edited

Loading