Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mempool improvements #6640

Merged
merged 161 commits into from
Dec 5, 2024
Merged

Mempool improvements #6640

merged 161 commits into from
Dec 5, 2024

Conversation

andreibancioiu
Copy link
Collaborator

@andreibancioiu andreibancioiu commented Nov 28, 2024 •
edited
Loading

Reasoning behind the pull request

  • Transactions selection (from mempool) is sub-optimal: sorting is sub-optimal, while the volume of selected transactions isn't wisely measured / capped (and, in conjunction with the front-running protection, this sometimes produces undesirable effects - blocks not being full).
  • Scheduled processing interacts with the transactions selection & nonce notification, and leads to sub-optimal processing of transactions (and blocks not being full).

Proposed changes

Now, transactions are selected with respect to their price per gas unit. Furthermore, selection is capped at a specific amount of gas. Not-executable transactions are excluded from selection (best-effort). Nonce gaps (and other anomalies) aren't detected using the notifications-based mechanism anymore. Instead, the mempool has direct access to query the blockchain state during selection.

Testing procedure

  • Standard testing
  • Scenarios (private)
  • Pre-release testing on a few validators, on mainnet.

Pre-requisites

Based on the Contributing Guidelines the PR author and the reviewers must check the following requirements are met:

  • was the PR targeted to the correct branch?
  • if this is a larger feature that probably needs more than one PR, is there a feat branch created?
  • if this is a feat branch merging, do all satellite projects have a proper tag inside go.mod?

andreibancioiu and others added 30 commits July 10, 2024 00:10
@andreibancioiu
Reference newer storage-go.
859fd8d
@andreibancioiu
Fix tests.
10c67f1
@andreibancioiu
Newer storage-go.
fd214d6
@andreibancioiu
Adjust NumTxPerSenderBatchForFillingMiniblock, MaxGasBandwidthPerBatc…
8f92b0c 
…hPerSender.
@andreibancioiu
Update go.mod.
3ea4345
@andreibancioiu
Fix tests.
3b2f40e
@andreibancioiu
Integrate new selection parameter.
1202873
@andreibancioiu
Adjust / skip test.
fea4455
@andreibancioiu
Some TODOs, update reference of storage-go.
61efec0
@andreibancioiu
Optimize shardedTxPool.removeTxBulk.
81e742a
@andreibancioiu
Testing workaround - notify account nonce when adding tx in pool.
6e7d92d
@andreibancioiu
Update reference.
17aa534
@andreibancioiu
Extra check on workaround for testing.
618f2f0
@andreibancioiu
Pass epoch notifier to sharded TX pool.
04a2764
@cristure
calculate hash only on log trace level set.
6841919
@cristure
add non-memory thrasher test.
85d93ac
@andreibancioiu
Minor refactoring / cleanup.
3a2a25a
@andreibancioiu
Define account nonce provider. Pass epoch notifier and account nonce …
20f174f 
…provider to sharded tx pool.
@andreibancioiu
Fix some tests.
652d417
@andreibancioiu
In sharded tx pool, on tx add, notify cache about sender nonce.
9063904
@andreibancioiu
Fix additional tests.
c8f4d6d
@andreibancioiu
Fix short test.
f126716
@andreibancioiu
Fix linter.
8a4f344
@andreibancioiu
Fix long tests (return err in nonce provider).
416ad59
@andreibancioiu
Adjust log, tests. Plus minor refactoring.
25cd44f
@andreibancioiu
Merge branch 'master' into master-into-mempool-2024-09-08
8bde909
@andreibancioiu
Merge branch 'master' into mempool-10
ce579e3
@andreibancioiu
Merge pull request #6381 from multiversx/master-into-mempool-2024-09-08
c48f5d2 
Master into feat/mempool
@andreibancioiu
Add TODO.
665c53c
@andreibancioiu
Merge branch 'master' into mempool-10
2d641f5
@multiversx multiversx deleted a comment from github-actions bot Nov 29, 2024
@multiversx multiversx deleted a comment from github-actions bot Nov 29, 2024
@multiversx multiversx deleted a comment from github-actions bot Nov 29, 2024
@andreibancioiu andreibancioiu mentioned this pull request Nov 29, 2024
Merged
@sstanculeanu
Copy link
Collaborator

Run Tests:
mx-chain-simulator-go: main
mx-chain-testing-suite: main

@github-actions GitHub Actions
Copy link

📊 MultiversX Automated Test Report: View Report

🔄 Build Details:

  • mx-chain-go Commit Hash: a7c336eb8b618397f8576c0c0108fcd4ad8038f1
  • Current Branch: feat/mempool
  • mx-chain-go Target Branch: rc/spica-patch-mempool
  • mx-chain-simulator-go Target Branch: main
  • mx-chain-testing-suite Target Branch: main

🚀 Environment Variables:

  • TIMESTAMP: 29112024-164138
  • PYTEST_EXIT_CODE: 0
    🎉 MultiversX CI/CD Workflow Complete!

@github-actions GitHub Actions
Copy link

📊 MultiversX Automated Test Report: View Report

🔄 Build Details:

  • mx-chain-go Commit Hash: 4c24359ab89654bee5b184d2506d31cd959165bf
  • Current Branch: master
  • mx-chain-go Target Branch: ``
  • mx-chain-simulator-go Target Branch: main
  • mx-chain-testing-suite Target Branch: main

🚀 Environment Variables:

  • TIMESTAMP: 29112024-164158
  • PYTEST_EXIT_CODE: 1
    🎉 MultiversX CI/CD Workflow Complete!

@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 2, 2024

⚠️ No report was generated due to an error or cancellation of the process.
Please checkout gh action logs for details

@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 2, 2024

⚠️ No report was generated due to an error or cancellation of the process.
Please checkout gh action logs for details

@andreibancioiu andreibancioiu mentioned this pull request Dec 2, 2024
Closed
@andreibancioiu
Copy link
Collaborator Author

Run Tests:
mx-chain-simulator-go: main
mx-chain-testing-suite: main

@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 2, 2024

📊 MultiversX Automated Test Report: View Report

🔄 Build Details:

  • mx-chain-go Commit Hash: 4c24359ab89654bee5b184d2506d31cd959165bf
  • Current Branch: master
  • mx-chain-go Target Branch: ``
  • mx-chain-simulator-go Target Branch: main
  • mx-chain-testing-suite Target Branch: main

🚀 Environment Variables:

  • TIMESTAMP: 02122024-111600
  • PYTEST_EXIT_CODE: 0
    🎉 MultiversX CI/CD Workflow Complete!

@andreibancioiu
Copy link
Collaborator Author

Run Tests:
mx-chain-simulator-go: mempool
mx-chain-testing-suite: main

@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 4, 2024

⚠️ No report was generated due to an error or cancellation of the process.
Please checkout gh action logs for details

@andreibancioiu andreibancioiu marked this pull request as ready for review December 4, 2024 16:17
@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 4, 2024

📊 MultiversX Automated Test Report: View Report

🔄 Build Details:

  • mx-chain-go Commit Hash: 11fc7c626218d0d8e7ea4b72d8ca524e27ac22e9
  • Current Branch: feat/mempool
  • mx-chain-go Target Branch: rc/spica-patch-mempool
  • mx-chain-simulator-go Target Branch: mempool
  • mx-chain-testing-suite Target Branch: main

🚀 Environment Variables:

  • TIMESTAMP: 04122024-162505
  • PYTEST_EXIT_CODE: 0
    🎉 MultiversX CI/CD Workflow Complete!

@andreibancioiu andreibancioiu merged commit 7c4667b into rc/spica-patch-mempool Dec 5, 2024
9 checks passed
@andreibancioiu andreibancioiu deleted the feat/mempool branch December 5, 2024 10:12
@andreibancioiu andreibancioiu mentioned this pull request Dec 5, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@btc-fan btc-fan btc-fan approved these changes

@AdoAdoAdo AdoAdoAdo AdoAdoAdo approved these changes

@sstanculeanu sstanculeanu sstanculeanu approved these changes

Assignees

@andreibancioiu andreibancioiu

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@andreibancioiu @sstanculeanu @AdoAdoAdo @btc-fan @cristure