Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GD Optimizations #8

Closed
wants to merge 91 commits into from
Closed

GD Optimizations #8

wants to merge 91 commits into from

Commits on Jun 26, 2024

  1. Refactor pruning proof validation to many functions

    Co-authored-by: Ori Newman <[email protected]>
    coderofstuff and someone235 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    5d3b10f View commit details
    Browse the repository at this point in the history
  2. Use blue score as work for higher levels

    Co-authored-by: Ori Newman <[email protected]>
    coderofstuff and someone235 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    40f1cc9 View commit details
    Browse the repository at this point in the history
  3. Remove pruning processor dependency on gd managers

    Co-authored-by: Ori Newman <[email protected]>
    coderofstuff and someone235 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    4af6773 View commit details
    Browse the repository at this point in the history
  4. Consistency renaming

    Co-authored-by: Ori Newman <[email protected]>
    coderofstuff and someone235 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    1df5a22 View commit details
    Browse the repository at this point in the history
  5. Update db version

    Co-authored-by: Ori Newman <[email protected]>
    coderofstuff and someone235 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    d12592c View commit details
    Browse the repository at this point in the history
  6. GD Optimizations

    Co-authored-by: Ori Newman <[email protected]>
    coderofstuff and someone235 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    2bea765 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    902b217 View commit details
    Browse the repository at this point in the history
  8. Ensure parents are in relations; Add comments

    apply_proof only inserts parent entries for a header from the proof into
    the relations store for a level if there was GD data in the old stores
    for that header.
    
    This adds a check to filter out parent records not in relations store
    coderofstuff committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    7f1f412 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    f49478a View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    879c135 View commit details
    Browse the repository at this point in the history
  11. Relax the panic to warn when finished_headers and couldn't find suffi…

    …cient root
    
    This happens when there's not enough headers in the pruning proof but it satisfies validation
    coderofstuff committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    56c0b40 View commit details
    Browse the repository at this point in the history
  12. Error handling for gd on higher levels

    relations.get_parents on GD gets extra parents that aren't in the
    current GD store. so get_blue_work throws an error
    
    next, ORIGIN was mising from the GD so add that
    coderofstuff committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    43e9f9e View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    34f20ab View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    2654b25 View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    ba04929 View commit details
    Browse the repository at this point in the history
  16. Configuration menu
    Copy the full SHA
    a45b571 View commit details
    Browse the repository at this point in the history
  17. Configuration menu
    Copy the full SHA
    edb5cd3 View commit details
    Browse the repository at this point in the history
  18. Keep DB Version at 3

    The new prefixes added are compatible with the old version. We don't want to trigger a db delete with this change
    coderofstuff committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    e81394f View commit details
    Browse the repository at this point in the history
  19. Configuration menu
    Copy the full SHA
    0e8c788 View commit details
    Browse the repository at this point in the history
  20. remove simpa changes

    coderofstuff committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    56f9dab View commit details
    Browse the repository at this point in the history
  21. Remove rewriting origin to primary GD

    It's already on there
    coderofstuff committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    c5be8ad View commit details
    Browse the repository at this point in the history
  22. Configuration menu
    Copy the full SHA
    8d15e27 View commit details
    Browse the repository at this point in the history
  23. Lint fixes

    coderofstuff committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    1c6b585 View commit details
    Browse the repository at this point in the history
  24. Configuration menu
    Copy the full SHA
    273aa81 View commit details
    Browse the repository at this point in the history
  25. Address initial comments

    coderofstuff committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    f9b3fda View commit details
    Browse the repository at this point in the history
  26. Configuration menu
    Copy the full SHA
    ca8bb42 View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2024

  1. Add small safety margin to proof at level 0

    This prevents the case where new root is an anticone of old root
    coderofstuff committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    61183fa View commit details
    Browse the repository at this point in the history

Commits on Jun 29, 2024

  1. Configuration menu
    Copy the full SHA
    34bc88f View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    da1cfe3 View commit details
    Browse the repository at this point in the history

Commits on Jul 6, 2024

  1. Configuration menu
    Copy the full SHA
    a23d1dd View commit details
    Browse the repository at this point in the history
  2. Re-apply missed comment

    coderofstuff committed Jul 6, 2024
    Configuration menu
    Copy the full SHA
    974d200 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    6ea8328 View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2024

  1. Configuration menu
    Copy the full SHA
    f8baf69 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    4f6ad73 View commit details
    Browse the repository at this point in the history
  3. Various fixes

    - Keep using old root to minimize proof size. Old root is calculated
      using the temporary gd stores
    - fix the off-by-one in block_at_depth and chain_up_to_depth
    - revert the temp fix to sync with the off-by-one
    coderofstuff committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    bc56e65 View commit details
    Browse the repository at this point in the history

Commits on Jul 14, 2024

  1. Revert "Various fixes"

    This reverts commit bc56e65.
    
    This experimental commit requires a bit more thinking to apply, and
    optimization can be deferred.
    coderofstuff committed Jul 14, 2024
    Configuration menu
    Copy the full SHA
    efbb083 View commit details
    Browse the repository at this point in the history

Commits on Jul 16, 2024

  1. Revert better proof check

    Recreates the GD stores for the current consensus by checking existing proof
    coderofstuff committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    a585be7 View commit details
    Browse the repository at this point in the history
  2. Fix: use cc gd store

    coderofstuff committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    e7625c7 View commit details
    Browse the repository at this point in the history

Commits on Aug 18, 2024

  1. Configuration menu
    Copy the full SHA
    0741151 View commit details
    Browse the repository at this point in the history

Commits on Aug 20, 2024

  1. Configuration menu
    Copy the full SHA
    d48e049 View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2024

  1. Add trusted blocks to all relevant levels during apply_proof

    As opposed to applying only to level 0
    coderofstuff committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    89f1701 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    3e249f9 View commit details
    Browse the repository at this point in the history

Commits on Aug 24, 2024

  1. Configuration menu
    Copy the full SHA
    fb3d1e9 View commit details
    Browse the repository at this point in the history
  2. Explain finished headers logic

    Add back the panic if we couldn't find the required block and our headers are done
    
    Add explanation in comment for why trying anyway if finished_headers is acceptable
    coderofstuff committed Aug 24, 2024
    Configuration menu
    Copy the full SHA
    c9855d3 View commit details
    Browse the repository at this point in the history
  3. clarify comment

    coderofstuff committed Aug 24, 2024
    Configuration menu
    Copy the full SHA
    a63acdb View commit details
    Browse the repository at this point in the history
  4. Rename old_root to depth_based_root

    explain logic for the two root calculation
    coderofstuff committed Aug 24, 2024
    Configuration menu
    Copy the full SHA
    46dbac3 View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2024

  1. Configuration menu
    Copy the full SHA
    bdf205e View commit details
    Browse the repository at this point in the history
  2. More merge fixes

    coderofstuff committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    4c9f997 View commit details
    Browse the repository at this point in the history
  3. Merge pull request #9 from someone235/gd-proof-ignore-root-past

    When building pruning point proof ghostdag data, ignore blocks before root
    coderofstuff authored Aug 27, 2024
    Configuration menu
    Copy the full SHA
    42a3030 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    4f2aace View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2024

  1. Configuration menu
    Copy the full SHA
    bc2e648 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    56b4392 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2024

  1. Configuration menu
    Copy the full SHA
    66d4ebe View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    18158e6 View commit details
    Browse the repository at this point in the history

Commits on Sep 1, 2024

  1. Configuration menu
    Copy the full SHA
    8d042b2 View commit details
    Browse the repository at this point in the history

Commits on Sep 2, 2024

  1. Configuration menu
    Copy the full SHA
    09f1443 View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2024

  1. Configuration menu
    Copy the full SHA
    2913bda View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2024

  1. Configuration menu
    Copy the full SHA
    095a9f3 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    65948d3 View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2024

  1. Configuration menu
    Copy the full SHA
    6021a1e View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2024

  1. Configuration menu
    Copy the full SHA
    009f9a8 View commit details
    Browse the repository at this point in the history

Commits on Sep 29, 2024

  1. Configuration menu
    Copy the full SHA
    3bc2844 View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2024

  1. Bump tonic and prost versions, adapt middlewares (kaspanet#553)

    * bump tonic, prost versions
    update middlewares
    
    * use unbounded channel
    
    * change log level to trace
    
    * use bounded channel
    
    * reuse counts bytes body to measure bytes body
    
    * remove unneeded clone
    biryukovmaxim authored Sep 30, 2024
    Configuration menu
    Copy the full SHA
    66959d4 View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2024

  1. Configuration menu
    Copy the full SHA
    a60bd8c View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2024

  1. Configuration menu
    Copy the full SHA
    2b0f3ab View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    f1d6904 View commit details
    Browse the repository at this point in the history

Commits on Oct 6, 2024

  1. Configuration menu
    Copy the full SHA
    b37f030 View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2024

  1. replace statrs and statest deps & upgrade some deps. (kaspanet#425)

    * replace statrs and statest deps.
    
    * remove todo in toml.cargo and fmt & lints.
    
    * do a run of `cargo audit fix` for some miscellaneous reports.
    
    * use maintained alt ks crate.
    
    * add cargo.lock.
    
    * update
    
    * use new command
    
    * newline
    
    * refresh cargo lock with a few more version updates
    
    * fix minor readme glitches
    
    ---------
    
    Co-authored-by: Michael Sutton <[email protected]>
    D-Stacks and michaelsutton authored Oct 7, 2024
    Configuration menu
    Copy the full SHA
    1378e7b View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2024

  1. enhance tx inputs processing (kaspanet#495)

    * sighash reused trait
    
    * benches are implemented
    
    * use cache per iteration per function
    
    * fix par versions
    
    * fix benches
    
    * use upgreadable read
    
    * use concurrent cache
    
    * use hashcache
    
    * dont apply cache
    
    * rollback rwlock and indexmap.
    
    * remove scc
    
    * apply par iter to `check_scripts`
    
    * refactor check_scripts fn, fix tests
    
    * fix clippy
    
    * add bench with custom threadpool
    
    * style: fmt
    
    * suppress warnings
    
    * Merge branch 'master' into bcm-parallel-processing
    
    * renames + map err
    
    * reuse code
    
    * bench: avoid exposing cache map + iter pools in powers of 2
    
    * simplify check_sig_op_counts
    
    * use thread pool also if a single input
    1. to avoid confusion
    2. since tokio blocking threads are not meant to be used for processing anyway
    
    * remove todo
    
    * clear cache instead of recreate
    
    * use and_then (so map_err can be called in a single location)
    
    * extend check scripts tests for better coverage of the par_iter case
    
    ---------
    
    Co-authored-by: Michael Sutton <[email protected]>
    biryukovmaxim and michaelsutton authored Oct 8, 2024
    Configuration menu
    Copy the full SHA
    1274e9c View commit details
    Browse the repository at this point in the history

Commits on Oct 12, 2024

  1. Configuration menu
    Copy the full SHA
    362e564 View commit details
    Browse the repository at this point in the history
  2. Parallelize MuHash calculations (kaspanet#575)

    * Parallelize MuHash calculations
    
    MuHash calculations are additive and can be done in chunks then later combined
    
    * Reimplement validate tx with muhash as a separate fn
    
    * Use smallvec for muhash parallel
    
    Co-authored-by: Michael Sutton <[email protected]>
    
    * Add independent rayon order test
    
    * Filter some data
    
    * Use tuple_windows for test iter
    
    ---------
    
    Co-authored-by: Michael Sutton <[email protected]>
    coderofstuff and michaelsutton authored Oct 12, 2024
    Configuration menu
    Copy the full SHA
    c59a0d1 View commit details
    Browse the repository at this point in the history

Commits on Oct 13, 2024

  1. Muhash parallel reduce -- optimize U3072 mul when LHS = one (kaspanet…

    …#581)
    
    * semantic: add `from` ext methods
    
    * muhash from txs benchmark
    
    * optimization: in u3072 mul test if lhs is one
    
    * extract `parallelism_in_power_steps`
    
    * comment
    michaelsutton authored Oct 13, 2024
    Configuration menu
    Copy the full SHA
    0df2de5 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    7b7d659 View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2024

  1. Some cleanup

    - use BlueWorkType
    - fix some comments
    coderofstuff committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    2e8deab View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    4168456 View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2024

  1. Rust 1.82 fixes + mempool std sig op count check (kaspanet#583)

    * rust 1.82 fixes
    
    * sig op count std check
    michaelsutton authored Oct 22, 2024
    Configuration menu
    Copy the full SHA
    a40efbb View commit details
    Browse the repository at this point in the history
  2. typo(cli/utils): kaspa wording (kaspanet#582)

    Co-authored-by: Michael Sutton <[email protected]>
    IzioDev and michaelsutton authored Oct 22, 2024
    Configuration menu
    Copy the full SHA
    aac16a9 View commit details
    Browse the repository at this point in the history

Commits on Oct 26, 2024

  1. Configuration menu
    Copy the full SHA
    a11b135 View commit details
    Browse the repository at this point in the history
  2. Cleaner find_selected_parent_header_at_level

    Co-authored-by: Michael Sutton <[email protected]>
    coderofstuff and michaelsutton authored Oct 26, 2024
    Configuration menu
    Copy the full SHA
    b6af1af View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    dcd3f5c View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    32f890d View commit details
    Browse the repository at this point in the history
  5. Lint and logic fix

    coderofstuff committed Oct 26, 2024
    Configuration menu
    Copy the full SHA
    15486f8 View commit details
    Browse the repository at this point in the history

Commits on Oct 28, 2024

  1. Reduce loop depth increase on level proof retries

    Co-authored-by: Michael Sutton <[email protected]>
    coderofstuff and michaelsutton authored Oct 28, 2024
    Configuration menu
    Copy the full SHA
    6a41f26 View commit details
    Browse the repository at this point in the history
  2. Update consensus/src/processes/pruning_proof/mod.rs

    Co-authored-by: Michael Sutton <[email protected]>
    coderofstuff and michaelsutton authored Oct 28, 2024
    Configuration menu
    Copy the full SHA
    cf476f2 View commit details
    Browse the repository at this point in the history
  3. Comment cleanup

    coderofstuff committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    be105f9 View commit details
    Browse the repository at this point in the history
  4. Remove unnecessary clone

    Co-authored-by: Michael Sutton <[email protected]>
    coderofstuff and michaelsutton authored Oct 28, 2024
    Configuration menu
    Copy the full SHA
    59b1a05 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    5cd27f3 View commit details
    Browse the repository at this point in the history
  6. Cleaner reachability_stores type

    Co-authored-by: Michael Sutton <[email protected]>
    coderofstuff and michaelsutton authored Oct 28, 2024
    Configuration menu
    Copy the full SHA
    8e03452 View commit details
    Browse the repository at this point in the history

Commits on Oct 29, 2024

  1. Configuration menu
    Copy the full SHA
    01024b1 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    1777b80 View commit details
    Browse the repository at this point in the history
  3. A few minor leftovers

    michaelsutton committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    f83a70f View commit details
    Browse the repository at this point in the history