Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

peer, main, netsync, blockchain: parallel block downloads #2226

Open
wants to merge 11 commits into
base: master
Choose a base branch
from

Commits on Aug 20, 2024

  1. peer: make peer meet query.Peer interface

    query.Peer is used for downloading blocks out of order during headers
    first download.  Methods SubscribeRecvMsg() and OnDisconnect() are added
    to abide by the interface.
    kcalvinalvin committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    782dd2f View commit details
    Browse the repository at this point in the history
  2. main: add ConnectedPeers() to server

    ConnectedPeers returns all the currently connected peers.  This is used
    to provide the query.WorkManager with all the currently connected peers
    from the netsync package.
    kcalvinalvin committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    510083d View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2024

  1. netsync: add checkpointedBlocksQuery

    checkpointedBlocksQuery is a helper to create []*query.Request which can
    be passed off to query.Workmanager to query for wire.Messages to
    multiple peers.  This is useful for downloading blocks out of order from
    multiple peers during ibd.
    kcalvinalvin committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    eebcd60 View commit details
    Browse the repository at this point in the history
  2. netsync: refactor handleBlockMsg

    handleBlockMsg used to check that the block header is both valid and
    then process the blocks as they come in.  It's now refactored so that
    it also handles blocks that are not in order.  For out of order block
    downloads handleBlockMsg would mark the block as an orphan but it's now
    refactored to handle those cases.
    
    Whenever a block that's not the next from the chain tip is received,
    it's now temporarily stored in memory until the next block from the
    chain tip is received.  And then all the blocks that are in sequence are
    processed.
    kcalvinalvin committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    e3e450f View commit details
    Browse the repository at this point in the history
  3. netsync: add peerSubscription

    peerSubscription is added to Manager which will allow it subscribers to
    receive peers through the channel whenever the Manager is aware of a new
    peer that it's been connected to.  This is useful to alert
    query.Workmanager that a new peer that's been connected to is eligible
    to download blocks from.
    kcalvinalvin committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    28d9f2b View commit details
    Browse the repository at this point in the history
  4. netsync, main: add ConnectedPeers to Manager

    ConnectedPeers returns all the currently connected peers and any new
    peer that's additionally connected through the returned channel.  This
    method is required for query.Workmanager as it needs ot receive peers
    that it can request blocks from.
    kcalvinalvin committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    ee0f49d View commit details
    Browse the repository at this point in the history
  5. netsync: add fetchManager and request blocks from it

    The blocks that were requested from headers are now sent over to
    query.Workmanager where it will rank peers based on their speed and
    request blocks from them accordingly.  This allows for quicker block
    downloads as:
    1: Workmanager will prioritize faster peers.
    2: Workmanager is able to ask from multiple peers.
    kcalvinalvin committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    159cd2d View commit details
    Browse the repository at this point in the history
  6. blockchain: store block concurrently while verifying it

    Storing block happens before the block validation is done and this can
    be a bottleneck on computers with slow disks.  Allowing for concurrent
    block storage saves time as the disk operation can be done in parallel
    with the cpu operations of verifying the block.
    kcalvinalvin committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    ed15b8a View commit details
    Browse the repository at this point in the history
  7. netsync: add logger for blocks downloaded from different peers during

    headers-first block download
    kcalvinalvin committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    7c1b80f View commit details
    Browse the repository at this point in the history
  8. netsync: don't reset the requestedBlocks in headersFirst

    Resetting the requestedBlocks state in headersFirst is problematic since
    we may be banning peers that are still good.
    kcalvinalvin committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    d8d8dd7 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    2cbb562 View commit details
    Browse the repository at this point in the history