Code

Mempool improvements#62

Code

Merged

andreibancioiu merged 194 commits intorc/spica-patch-mempoolmultiversx/mx-chain-storage-go:rc/spica-patch-mempoolfrom feat/mempoolmultiversx/mx-chain-storage-go:feat/mempoolCopy head branch name to clipboardDec 5, 2024

+2,913-3,261

Commits

Commits on Jul 1, 2024

• On NotifyAccountNonce, remove transactions with lower nonces.

  andreibancioiu

  committed
  41fb6b8

Commits on Jul 3, 2024

• WrappedTransaction.computeFee() delegates to the Node's economicsData logic.

  andreibancioiu

  committed
  dc2ff1c

• Remove older feeComputationHelper.

  andreibancioiu

  committed
  1875775

• New score function (avg ppu, scaled, logistic, re-scaled). Refactoring.

  andreibancioiu

  committed
  5282c22

• Integrate new score params in txListForSender.

  andreibancioiu

  committed
  8504682

• Implement noncesTracker. Able to tell if a nonces sequence is "spotless" (no gaps, no duplicates). Sketch some tests.

  andreibancioiu

  committed
  3d04082

Commits on Jul 4, 2024

• Adjust tests.

  andreibancioiu

  committed
  ad6bc75

• Adjust log levels.

  andreibancioiu

  committed
  d3f9e63

• Adjust some tests.

  andreibancioiu

  committed
  3dd13fe

Commits on Jul 5, 2024

• Fix score, fix tests.

  andreibancioiu

  committed
  5874167

• Simplification of sender score params.

  andreibancioiu

  committed
  ccf6cf3

• Fix score, add tests.

  andreibancioiu

  committed
  7095682

• Optimizations.

  andreibancioiu

  committed
  21e1354

• Optimize "findInsertionPlace".

  andreibancioiu

  committed
  5e2d116

• Fix tests.

  andreibancioiu

  committed
  c795b7a

Commits on Jul 8, 2024

• Additional tests.

  andreibancioiu

  committed
  11ebe83

• Monitoring cleanup.

  andreibancioiu

  committed
  5fb4b11

• Compute score on tx addition / removal. Sort senders only on selection & eviction.

  andreibancioiu

  committed
  b7dd503

• Fix tests, benchmarks.

  andreibancioiu

  committed
  6f61e51

• Optimization at insertion time.

  andreibancioiu

  committed
  c1697a5

• Optimization at tx removal.

  andreibancioiu

  committed
  a2bc648

• Bit of refactoring, more tests.

  andreibancioiu

  committed
  377369b

Commits on Jul 10, 2024

• Monitor senders histogram.

  andreibancioiu

  committed
  610342c

• Additional logging.

  andreibancioiu

  committed
  ce5c433

• Batchsize, bandwidth computation.

  andreibancioiu

  committed
  4d835de

• Additional monitoring.

  andreibancioiu

  committed
  aec42e7

• Adjust printing.

  andreibancioiu

  committed
  14f8719

• computeSelectionSenderConstraints() based on score.

  andreibancioiu

  committed
  3557e7c

• Fix condition.

  andreibancioiu

  committed
  b1b22d2

• Rename constant, fix array length.

  andreibancioiu

  committed
  acdfd8d

• Renaming.

  andreibancioiu

  committed
  f996c68

Commits on Jul 11, 2024

• Some todos, notes.

  andreibancioiu

  committed
  46c0d40

• notifyAccountNonce: remove sender if empty after txs removal.

  andreibancioiu

  committed
  73b5848

• No more logic on "sweep sweepable senders". Old transactions are removed by Node's transactions cleaner. Senders with any gaps have score 0 - they are the last to select, first to evict.

  andreibancioiu

  committed
  9c234af

• Remove concept of "grace period" for senders with initial gaps. No transactions of a sender with initial gap are selected anymore. If this is an issue, sender has to re-broadcast - pool will get th…

  Show description for c5cb6fb
  andreibancioiu

  committed
  c5cb6fb

• Remove the counter numFailedSelections.

  andreibancioiu

  committed
  3a15954

• Minor refactoring.

  andreibancioiu

  committed
  1f4033a

• A little bit of refactoring.

  andreibancioiu

  committed
  ad8c55b

Commits on Jul 15, 2024

• Add total gas constraints when selecting transactions.

  andreibancioiu

  committed
  ba403bd

• Bit of refactoring.

  andreibancioiu

  committed
  e8e808d

• Shuffle the items of a group (senders with the same score). Add some benchmarks.

  andreibancioiu

  committed
  7a2e122

• Some refactoring.

  andreibancioiu

  committed
  0714177

• Fix linter, fix tests.

  andreibancioiu

  committed
  61d37d4

Commits on Jul 16, 2024

• Monitoring, refactoring, optimizations. Fix score computation.

  andreibancioiu

  committed
  e50fbe5

• Optimization on notifyAccountNonce.

  andreibancioiu

  committed
  4f7726d

Commits on Jul 17, 2024

• Adjust some tests.

  andreibancioiu

  committed
  1da6b62

• Fix build.

  andreibancioiu

  committed
  8920291

Commits on Jul 18, 2024

• Better diagnostics.

  andreibancioiu

  committed
  50fa20e

Commits on Jul 25, 2024

• Diagnose selection.

  andreibancioiu

  committed
  13a3a96

• Simplifications, refactoring.

  andreibancioiu

  committed
  ed6b638

• Monitoring simplifications.

  andreibancioiu

  committed
  9211efa

• Fix logs.

  andreibancioiu

  committed
  dede39c

Commits on Jul 26, 2024

• Extra logs, test, comments etc.

  andreibancioiu

  committed
  843bbd2

Commits on Jul 30, 2024

• Refactoring, fix interface etc.

  andreibancioiu

  committed
  9b8f619

Commits on Aug 6, 2024

• Fix diagnosis / duration.

  andreibancioiu

  committed
  07437b3

Commits on Sep 6, 2024

• Additional logging.

  andreibancioiu

  committed
  a03e955

• Additional logs when gaps are detected (at selection time).

  andreibancioiu

  committed
  91e0519

• Fix evictTransactionsWithLowerNonces.

  andreibancioiu

  committed
  2235ce3

Commits on Oct 2, 2024

• Sketch readme.

  andreibancioiu

  committed
  f79ad1b

• Docs, work in progress.

  andreibancioiu

  committed
  4a30d0c

• Docs, work in progress. Additional tests.

  andreibancioiu

  committed
  8352a3f

Commits on Oct 29, 2024

• On selection, don't use "numRequested" anymore (limit is wrt. "gas requested").

  andreibancioiu

  committed
  712f71a

• Preparatory cleanup.

  andreibancioiu

  committed
  a30472f

Commits on Oct 30, 2024

• Sketch selection using merges.

  andreibancioiu

  committed
  a71a512

• Sketch getTxsWithoutGaps (not tested).

  andreibancioiu

  committed
  a8c39b7

Commits on Oct 31, 2024

• Sketch computePricePerGasUnit.

  andreibancioiu

  committed
  5a42102

• Fix isTransactionGreater().

  andreibancioiu

  committed
  d1063be

• Sketch parallelization (not tested).

  andreibancioiu

  committed
  0d96e18

• Fix & simplify mergeTwoBunchesOfTransactions.

  andreibancioiu

  committed
  b8cf28a

• Integrate new selection (work in progress).

  andreibancioiu

  committed
  f40df76

• Sketch / fix merging.

  andreibancioiu

  committed
  5e67c02

• Improve isTransactionGreater().

  andreibancioiu

  committed
  ce28507

• Add some tests.

  andreibancioiu

  committed
  e50d419

• Sketch eviction (not tested).

  andreibancioiu

  committed
  fc80fe3

Commits on Nov 1, 2024

• Simplify / refactor.

  andreibancioiu

  committed
  430744f

• Optimize / simplify merging.

  andreibancioiu

  committed
  71de44f

• Fix eviction, add some tests.

  andreibancioiu

  committed
  9099c66

• Fix call of wg.Done().

  andreibancioiu

  committed
  688b985

• Tests, trial and error.

  andreibancioiu

  committed
  5fa3797

• Additional testing.

  andreibancioiu

  committed
  0b2e201

• selectTransactionsUsingHeap().

  andreibancioiu

  committed
  65d4953

• Refactor; isTransactionMoreDesirableByProtocol().

  andreibancioiu

  committed
  4a9fa71

• More unit tests.

  andreibancioiu

  committed
  5a3fb74

• More unit tests.

  andreibancioiu

  committed
  4b1ec6d

• Refactor. Split min/max heaps.

  andreibancioiu

  committed
  8d28cc6

• Fix config, fix tests.

  andreibancioiu

  committed
  7565d6e

• Reimplement eviction, using min heap.

  andreibancioiu

  committed
  a6faf2e

• Fix eviction, work in progress.

  andreibancioiu

  committed
  1920a2f

• Remove selection using merges / merge sort.

  andreibancioiu

  committed
  608bda0

Commits on Nov 4, 2024

• Refactor, fix tests.

  andreibancioiu

  committed
  786f4e2

• Fix disabled cache.

  andreibancioiu

  committed
  8b89f4c

• Use float for PPU.

  andreibancioiu

  committed
  81bb99d

• Additional guard for gas limit.

  andreibancioiu

  committed
  0681184

• Refactor, simplify monitoring.

  andreibancioiu

  committed
  15c28eb

• Rename file.

  andreibancioiu

  committed
  62c0f96

• Refactor, fix logging.

  andreibancioiu

  committed
  8348e11

• Additional logging.

  andreibancioiu

  committed
  4240502

• Fix selection (accumulated gas condition).

  andreibancioiu

  committed
  be56be9

• Log accumulated gas.

  andreibancioiu

  committed
  ff89596

• Fix detection of initial gaps.

  andreibancioiu

  committed
  e0332d2

• Cleanup, logging, benchmarks, minor optimizations.

  andreibancioiu

  committed
  8f53bbf

• Refactor / split file.

  andreibancioiu

  committed
  9b2b959

• Optimize isTransactionMoreDesirableByProtocol().

  andreibancioiu

  committed
  105701d

Commits on Nov 5, 2024

• Limit selection by "maxNum" transactions.

  andreibancioiu

  committed
  0257f64

• Additional tests and benchmarks.

  andreibancioiu

  committed
  3d7c01c

• Move tests.

  andreibancioiu

  committed
  7ee693c

• Fix disabled cache.

  andreibancioiu

  committed
  6a69b3e

• Fix lint warnings.

  andreibancioiu

  committed
  7c5b85d

• Remove unused constant (fix linter).

  andreibancioiu

  committed
  8e3d20f

• Minor simplification at eviction. Add function getTxsReversed().

  andreibancioiu

  committed
  2bc824a

• Cleanup, some readme, refactoring.

  andreibancioiu

  committed
  74bc040

• Cleanup.

  andreibancioiu

  committed
  5ee99a2

• Fix after self-review.

  andreibancioiu

  committed
  09f30ed

Commits on Nov 6, 2024

• Adjust logging.

  andreibancioiu

  committed
  9e31b00

Commits on Nov 7, 2024

• Merge branch 'rc/v1.7.next1' into selection-by-ppu

  andreibancioiu

  authored
  e8cf425

Commits on Nov 10, 2024

• Fix after review.

  andreibancioiu

  committed
  1a7e9f2

• In "getSequentialTxs", handle duplicates, as well.

  andreibancioiu

  committed
  b45ce82

Commits on Nov 11, 2024

• On nonce notifications, don't remove transactions. On RemoveTxByHash, remove those with lower or equal nonces, as well.

  andreibancioiu

  committed
  fe86b68

Commits on Nov 12, 2024

• Fix handling of lower nonces (act only when nonce is known). Refactor, rename, add tests.

  andreibancioiu

  committed
  44d940a

• Implement ForgetAllAccountNonces().

  andreibancioiu

  committed
  53c5bd8

Commits on Nov 13, 2024

• Describe selection flow in readme.

  andreibancioiu

  committed
  b91be71

• Typos.

  andreibancioiu

  committed
  10f1c36

• Update readme.

  andreibancioiu

  committed
  8cd2eb0

• Fix after review.

  andreibancioiu

  committed
  fa41478

• Merge pull request #37 from multiversx/rc/v1.7.next1

  Show description for a8324c5
  sstanculeanu

  authored
  a8324c5

• Don't notify about account nonces, don't forget etc.

  andreibancioiu

  committed
  2231c3d

• When selecting transactions, receive an account nonce provider.

  andreibancioiu

  committed
  b22177e

• Refactor, pass "AccountNonceProvider" in constructor.

  andreibancioiu

  committed
  3714684

• Quick sketch of selection changes.

  andreibancioiu

  committed
  74e7ac5

• Fix tests and comments.

  andreibancioiu

  committed
  d583eae

• Refactoring.

  andreibancioiu

  committed
  94131c7

• Refactor, optimize.

  andreibancioiu

  committed
  4ca46fd

Commits on Nov 14, 2024

• Fix after self-review.

  andreibancioiu

  committed
  20d4ba4

• Fix after review, drop logic around fnv32.

  andreibancioiu

  committed
  68835d1

• Update readme.

  andreibancioiu

  committed
  ee6476d

• Merge branch 'selection-by-ppu' into MX-16107-no-more-notify

  andreibancioiu

  committed
  4f94f63

• Additional tests.

  andreibancioiu

  committed
  0f01e5c

• Fix after review.

  andreibancioiu

  committed
  80b4979

Commits on Nov 15, 2024

• Receive the nonce provider in SelectTransactions, instead of receiving it in constructor. This way, we can receive the "real" processing accounts adapter (not the slower API accounts adapter).

  andreibancioiu

  committed
  f97351c

• Break the selection loop if it takes too long.

  andreibancioiu

  committed
  0db9d39

Commits on Nov 18, 2024

• AccountNonceProvider becomes AccountStateProvider (more information from account state is necessary).

  andreibancioiu

  committed
  3bbf408

• Additional logs on cross tx cache.

  andreibancioiu

  committed
  bae6b43

• Hold fee on tx, handle accumulated fees. Avoid non-executable transactions on missing balance for fees. Refactor.

  andreibancioiu

  committed
  613f5ba

• Reference new core-go.

  andreibancioiu

  committed
  95a888a

• Handle guarded transactions with same nonce.

  andreibancioiu

  committed
  ebe0e12

• Better readme etc.

  andreibancioiu

  committed
  c61ce2a

Commits on Nov 19, 2024

• Fix tests.

  andreibancioiu

  committed
  26f0189

Commits on Nov 20, 2024

• Better handling of not-executable transactions.

  andreibancioiu

  committed
  03c5adb

• A few optimizations.

  andreibancioiu

  committed
  679a465

• Fix fee exceeded balance detection. Refactoring.

  andreibancioiu

  committed
  de2620b

• Additional unit tests.

  andreibancioiu

  committed
  fe1bd09

• Adjust benchmark output.

  andreibancioiu

  committed
  67c1c6e

• Fix tests.

  andreibancioiu

  committed
  e20d2e5

• Additional unit tests.

  andreibancioiu

  committed
  b2fa1ce

• Fix after self review.

  andreibancioiu

  committed
  48853fb

• Fix after self review.

  andreibancioiu

  committed
  f159f60

• Remove out-of-reality test.

  andreibancioiu

  committed
  d5ef41b

Commits on Nov 21, 2024

• Update readme.

  andreibancioiu

  committed
  457c06c

• Merge pull request #55 from multiversx/selection-by-ppu

  Show description for 1229a5f
  andreibancioiu

  authored
  1229a5f

• Merge branch 'feat/mempool' into MX-16107-no-more-notify

  andreibancioiu

  authored
  1b8274d

• Fix linter issues.

  andreibancioiu

  committed
  92d0476

Commits on Nov 24, 2024

• Fix after review (part 1).

  andreibancioiu

  committed
  93c958a

• Call "isBadlyGuarded()" of the node.

  andreibancioiu

  committed
  65f6207

Commits on Nov 25, 2024

• Rename "AccountStateProvider" to "SelectionSession".

  andreibancioiu

  committed
  42abafc

• Additional tests.

  andreibancioiu

  committed
  3a7e727

• Handle consumed balance.

  andreibancioiu

  committed
  afd51e2

• Additional unit tests.

  andreibancioiu

  committed
  00adab2

Commits on Nov 26, 2024

• Delayed commit.

  andreibancioiu

  committed
  0710e1b

• Fix after review (part 2).

  andreibancioiu

  committed
  8a0cb50

Commits on Nov 27, 2024

• Merge branch 'MX-16107-no-more-notify' into MX-16179-guardians

  andreibancioiu

  committed
  edbfc56

• Fix test.

  andreibancioiu

  committed
  d1b4205

• Merge branch 'MX-16179-guardians' into MX-16160-consumed-balance

  andreibancioiu

  committed
  cde9ed9

• Merge pull request #57 from multiversx/MX-16107-no-more-notify

  Show description for 37ace93
  andreibancioiu

  authored
  37ace93

• Merge branch 'feat/mempool' into MX-16179-guardians

  andreibancioiu

  authored
  679f656

Commits on Nov 28, 2024

• Fix after review.

  andreibancioiu

  committed
  90581ee

• Merge pull request #58 from multiversx/MX-16179-guardians

  Show description for c8b1efb
  andreibancioiu

  authored
  c8b1efb

• Merge branch 'feat/mempool' into MX-16160-consumed-balance

  andreibancioiu

  committed
  156b124

• Merge pull request #59 from multiversx/MX-16160-consumed-balance

  Show description for a3a849d
  AdoAdoAdo

  authored
  a3a849d

• Rename "TxGasHandler" to "MempoolHost".

  andreibancioiu

  committed
  77bbd7a

• Move "GetTransferredValue()" to "MempoolHost". Define transferred value as a "precomputed" field.

  andreibancioiu

  committed
  ea3ee2d

Commits on Nov 29, 2024

• Update description of isTransactionMoreValuableForNetwork.

  andreibancioiu

  committed
  a019fe1

• Fix typo.

  andreibancioiu

  committed
  d11b529

• Merge pull request #60 from multiversx/MX-16211-optimize-get-value

  Show description for da4eff4
  andreibancioiu

  authored
  da4eff4

• Merge branch 'feat/mempool' into fix-txcache-readme

  andreibancioiu

  authored
  5a5a12b

• Merge pull request #61 from multiversx/fix-txcache-readme

  Show description for 3182661
  andreibancioiu

  authored
  3182661

• Merge branch 'main' into mempool-update-2024-11-29

  andreibancioiu

  committed
  d97a6ba

• Fix linter issues.

  andreibancioiu

  committed
  d0d8ce4

• Merge pull request #63 from multiversx/mempool-update-2024-11-29

  Show description for 8c5c73c
  andreibancioiu

  authored
  8c5c73c

Commits on Dec 2, 2024

• Add benchmarks for addition in worst case.

  andreibancioiu

  committed
  68d174a

• Fix linter issue.

  andreibancioiu

  committed
  f9a268f

• Merge pull request #64 from multiversx/MX-16202-sender-constraints

  Show description for 8d4a698
  andreibancioiu

  authored
  8d4a698

• Adjust benchmarks, lower maxNum when selecting transactions.

  andreibancioiu

  committed
  729f45b

• Adjust test config.

  andreibancioiu

  committed
  f8323f4

• Merge pull request #65 from multiversx/MX-16207-selection-max-num

  Show description for 16da4a5
  andreibancioiu

  authored
  16da4a5