add(mempool): Verify orphaned mempool transactions #8857

arya2 · 2024-09-09T19:45:37Z

Motivation

We want to verify transactions in the mempool that spend the transparent outputs of other transactions in the mempool.

Closes #8777.

Solution

Adds a handle to the mempool service in the transaction verifier
Adds AwaitOutput request in the mempool service
Updates the spent_utxos() method in the transaction verifier to try querying the mempool for spent outpoints before returning a TransparentInputNotFound error.
Polls the mempool service from the transaction verifier after verifying a mempool transaction that creates new UTXOs (in case they're spent by other mempool transactions that are still being verified)
Updates getblocktemplate method and ZIP-317 transaction selection to include transactions that spent the outputs of other transactions in the mempool when all of their dependencies have been selected

Tests

Added tests to check that:

Inserting a transaction also updates the transaction dependencies tracked in the verified set
The Storage.remove() method works as expected (the rest of evict_one() should be covered already)
The mempool responds to AwaitOutput requests when the output for the queried outpoint is:
- Added to the mempool
- Already available in the mempool
The transaction verifier:
- Calls the mempool with AwaitOutput requests as expected and processes UnspentOutput responses correctly
- Polls the mempool after verifying a mempool transaction that creates new transparent outputs
- Includes all of the expected spent_mempool_outpoints in its response
Long poll ids remaining consistent as long the mempool contents are unchanged (and everything else is unchanged)
ZIP-317 mempool transaction selection excludes transactions that have dependencies unless their dependencies have already been selected

If needed, this PR could also add tests for:

ZIP-317 mempool transaction selection always returns the right order of transactions (this could be unnecessary since they’re sorted afterwards anyway but it could be a good sanity check)
The getblocktemplate method returns transactions with dependencies in the right order when the transaction hashes would have them sorted in the wrong order
Ideally at least a basic acceptance test that checks that it all works via RPCs

Follow Up Work

The mempool's Downloads sends a result to the response channel too early, it should wait until the transaction is being inserted into the mempool's verified set.

PR Author's Checklist

The PR name will make sense to users.
The PR provides a CHANGELOG summary.
The solution is tested.
The documentation is up to date.
The PR has a priority label.

PR Reviewer's Checklist

The PR Author's checklist is complete.
The PR resolves the issue.

… mempool setup argument, adds and uses an `init_test()` fn for passing a closed channel receiver in tests where no mempool service is needed in the transaction verifier.

…nd a `new_for_tests()` constructor for convenience)

…t receiver, updates tests.

…ol service handle.

…dSet` `transactions()` return type.

…om the mempool's verified set

…onse

…m the tx verifier response

…ion outputs

…nsparent outputs has been verified. adds a TODO for adding a `pending_outputs` field to the mempool Storage

…ng outputs requests when inserting new transactions into the mempool's verified set

…ified set

… the caller handle it (required to avoid moving zip317 tx selection code to mempool)

…nless their dependencies have already been added.

zebra-rpc/src/methods/get_block_template_rpcs/zip317.rs

zebra-node-services/src/mempool.rs

zebra-consensus/src/transaction.rs

conradoplg

I'm still reviewing but I added a comment

conradoplg · 2024-10-11T14:55:48Z

zebrad/src/components/mempool/storage.rs

-    tip_rejected_exact: HashMap<UnminedTxId, ExactTipRejectionError>,
+    tip_rejected_exact: HashMap<transaction::Hash, ExactTipRejectionError>,


What is the reason for this change? The semantics of tip_rejected_exact is for it to incude transactions that have failed to validate e.g. due to an invalid signature. If this is changed to a Hash, then a transaction with e.g. the same data but a valid signature would be rejected when it shouldn't.

What is the reason for this change?

I think the broader change to mined ids was to avoid iterating through the verified set to find the auth digest when adding transaction dependencies (since OutPoint only includes the mined id).

The change to tip_rejected_exact was a mistake, it would work fine for the sendrawtransaction RPC, but I see it would incorrectly reject gossiped transaction ids when it should try to download and verify them.

Restored its type to the full UnminedTxId in ce6d169.

conradoplg · 2024-10-11T14:59:14Z

zebrad/src/components/mempool/storage.rs

+    pub fn insert(
+        &mut self,
+        tx: VerifiedUnminedTx,
+        spent_mempool_outpoints: Vec<transparent::OutPoint>,


Please document the new argument

Thank you for catching that, documented in cc76b40.

…omment

zebrad/src/components/mempool/storage/verified_set.rs

…thod without calling it.

… mined onto the best chain.

arya2 added 16 commits September 3, 2024 22:25

Adds a parameter to zebra_consensus::router::init() for accepting a…

fde3370

… mempool setup argument, adds and uses an `init_test()` fn for passing a closed channel receiver in tests where no mempool service is needed in the transaction verifier.

Adds a mempool argument to the transaction::Verifier constructor (a…

85fdd88

…nd a `new_for_tests()` constructor for convenience)

Removes Clone impl on transaction::Verifier to add mempool onesho…

174f4ea

…t receiver, updates tests.

Adds TODOs

3c05b03

updates transaction verifier's poll_ready() method to setup the mempo…

77aa2e3

…ol service handle.

Updates VerifiedSet struct used in mempool storage

5c7f45b

Updates mempool service and its Storage to use the updated `Verifie…

ecc2ddd

…dSet` `transactions()` return type.

updates created_outputs when inserting or removing a transaction fr…

e60197c

…om the mempool's verified set

Adds a TODO, updates field docs

745e6a7

Updates spent_utxos() to query the mempool for unspent outputs

c9c0a11

Adds spent_mempool_outpoints as a field on tx verifier mempool resp…

cbb34e6

…onse

Updates mempool Downloads to return the spent_mempool_outpoints fro…

be6fd7b

…m the tx verifier response

Updates Storage.insert() to accept a list of spent mempool transact…

90812b1

…ion outputs

Adds transaction dependencies when inserting a tx in VerifiedSet

d0b2d41

polls mempool svc from tx verifier when a mempool tx that creates tra…

0a72b41

…nsparent outputs has been verified. adds a TODO for adding a `pending_outputs` field to the mempool Storage

Adds pending_outputs field on mempool Storage and responds to pendi…

2fc7829

…ng outputs requests when inserting new transactions into the mempool's verified set

arya2 added A-mempool Area: Memory pool transactions P-Medium ⚡ labels Sep 9, 2024

arya2 self-assigned this Sep 9, 2024

github-actions bot added C-trivial Category: A trivial change that is not worth mentioning in the CHANGELOG labels Sep 9, 2024

arya2 added 9 commits September 16, 2024 16:07

replaces UnminedTxId type with transaction::Hash in mempool's ver…

ae87a27

…ified set

prune pending outputs when rejecting and removing same effects.

61e6e21

Remove dependent transactions from verified set when removing a tx

bebe50e

updates tests

29654b4

appeases clippy.

aed637a

removes unused len() method

a838934

fixes doc links

695a258

Adds transaction dependencies to the FullTransactions response, let…

bc981c0

… the caller handle it (required to avoid moving zip317 tx selection code to mempool)

updates block template construction to avoid including transactions u…

520660f

…nless their dependencies have already been added.

arya2 requested review from a team as code owners September 27, 2024 01:10

arya2 requested review from oxarbitrage and removed request for a team September 27, 2024 01:10

arya2 added A-consensus Area: Consensus rule updates A-concurrency Area: Async code, needs extra work to make it work properly. labels Sep 27, 2024

arya2 commented Sep 27, 2024

View reviewed changes

zebra-rpc/src/methods/get_block_template_rpcs/zip317.rs Outdated Show resolved Hide resolved

fixes zip317 block construction issue

66b841e

arya2 requested a review from upbqdn September 30, 2024 22:04

Fixes vectors test

2342d75

arya2 commented Oct 1, 2024

View reviewed changes

zebra-node-services/src/mempool.rs Outdated Show resolved Hide resolved

Update zebra-node-services/src/mempool.rs

56f2ad4

arya2 mentioned this pull request Oct 9, 2024

change(mempool): Return verification result after attempting to insert transactions in the mempool #8901

Draft

7 tasks

arya2 commented Oct 9, 2024

View reviewed changes

zebra-consensus/src/transaction.rs Outdated Show resolved Hide resolved

mpguerra requested review from conradoplg and removed request for a team, oxarbitrage and upbqdn October 10, 2024 10:35

oxarbitrage added the do-not-merge Tells Mergify not to merge this PR label Oct 10, 2024

conradoplg reviewed Oct 11, 2024

View reviewed changes

arya2 added 3 commits October 15, 2024 22:26

restores tip_rejected_exact type

ce6d169

updates affected tests

f897489

Documents the new argument in Storage::insert(), updates outdated c…

cc76b40

…omment

arya2 commented Oct 16, 2024

View reviewed changes

zebrad/src/components/mempool/storage/verified_set.rs Outdated Show resolved Hide resolved

Update zebrad/src/components/mempool/storage/verified_set.rs

64edb36

arya2 mentioned this pull request Oct 16, 2024

bug: Respond to getblockchaininfo requests when the state is empty #8939

Open

upbqdn and others added 3 commits October 18, 2024 19:56

Merge branch 'main' into verify-orphaned-mempool-txs

b11a708

fixes potential issue with calling buffered mempool's poll_ready() me…

a6d963f

…thod without calling it.

Avoids removing dependent transactions of transactions that have been…

5793ada

… mined onto the best chain.

arya2 mentioned this pull request Oct 19, 2024

change(rpc): Avoid re-verifying transactions in blocks if those transactions are in the mempool #8951

Draft

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add(mempool): Verify orphaned mempool transactions #8857

add(mempool): Verify orphaned mempool transactions #8857

arya2 commented Sep 9, 2024 •

edited

Loading

conradoplg left a comment

conradoplg Oct 11, 2024

arya2 Oct 16, 2024

conradoplg Oct 11, 2024

arya2 Oct 16, 2024

		tip_rejected_exact: HashMap<UnminedTxId, ExactTipRejectionError>,
		tip_rejected_exact: HashMap<transaction::Hash, ExactTipRejectionError>,

add(mempool): Verify orphaned mempool transactions #8857

Are you sure you want to change the base?

add(mempool): Verify orphaned mempool transactions #8857

Conversation

arya2 commented Sep 9, 2024 • edited Loading

Motivation

Solution

Tests

Follow Up Work

PR Author's Checklist

PR Reviewer's Checklist

conradoplg left a comment

Choose a reason for hiding this comment

conradoplg Oct 11, 2024

Choose a reason for hiding this comment

arya2 Oct 16, 2024

Choose a reason for hiding this comment

conradoplg Oct 11, 2024

Choose a reason for hiding this comment

arya2 Oct 16, 2024

Choose a reason for hiding this comment

arya2 commented Sep 9, 2024 •

edited

Loading