Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🔷 [Project Tracking] Resharding v3 #11881

Open
38 of 71 tasks
wacban opened this issue Aug 5, 2024 · 2 comments
Open
38 of 71 tasks

🔷 [Project Tracking] Resharding v3 #11881

wacban opened this issue Aug 5, 2024 · 2 comments
Assignees
@wacban
Labels
Near Core T-core Team: issues relevant to the core team

Comments

@wacban
Copy link
Contributor

wacban commented Aug 5, 2024 •
edited
Loading

Goals

  • The ability to increase the blockchain capacity by increasing the number of shards
  • The ability to balance the load across shards
  • The ability to do the above fast
  • The ability to do the above automatically (dynamic resharding)

References

Resharding Docs
Resharding V2 tracking issue

Tasks

Design - Instant Resharding

Design - Continuous Balancing @Longarithm & @Trisfald

  • High level design
  • Design document WIP
    • doc
    • How to handle large accounts?
      • account transfer
      • balancing manager should prevent large / busy accounts from hopping back and forth
    • Estimate and address the balancing overhead.
    • Account availability during transfer.
    • How to handle all sorts of receipts?
    • Estimate impact on the archival node storage.
    • Consider implementing a limit on account size.
    • Polish and verify design assumptions

Design - Resharding v2.1 @wacban

The old resharding is not particularly interesting because it takes at least an epoch to take effect. It's here mostly for completeness but it may be also worth considering it as a milestone towards the v3. Viability depends on the overlap with the end goal design and the amount of throw away work.

  • High level design, Short Epochs
  • Suitability as milestone towards Instant Resharding

Design - Automatic Scheduling aka Dynamic Resharding

Design - Mid-Epoch / Next-Epoch Resharding

The typical resharding operations, splitting and merging shards, require changes to the validator assignment. In order to execute resharding fast we will need to enhance the assignment to be able to be adjusted without the usual two epoch delay.

  • High level design
    • doc
    • Some ideas were presented in the Instant Resharding doc
    • Mid-Epoch may not work well with State Sync :(
@wacban wacban self-assigned this Aug 5, 2024
@wacban wacban added T-core Team: issues relevant to the core team Near Core labels Aug 5, 2024
@wacban wacban changed the title 🔷 [Tracking issue] Resharding v3 🔷 [Project Tracking] Resharding v3 Aug 6, 2024
@github-actions github-actions bot mentioned this issue Sep 1, 2024
Open
@wacban
Copy link
Contributor Author

wacban commented Sep 2, 2024

The design phase is now completed. We will be implementing the Instant Resharding approach. I will rearrange the issue to reflect the current work and I'll move the design task list down the description.

@walnut-the-cat
Copy link
Contributor

walnut-the-cat commented Sep 16, 2024 •
edited
Loading

Sept 2-13

@Trisfald Trisfald mentioned this issue Sep 30, 2024
Closed
33 tasks
@wacban wacban mentioned this issue Oct 28, 2024
Open
@wacban wacban mentioned this issue Jan 24, 2025
Merged
2 tasks
github-merge-queue bot pushed a commit that referenced this issue Jan 24, 2025
@wacban
feat(resharding) - Stabilize Resharding V3 (#12792)
09022ca 
# Feature to stabilize

This PR stabilizes Resharding V3. It introduces a new implementation for
resharding and two new shard layouts for the production networks.

# Context

- NEP: near/NEPs#568
- Implementation: #11881

# Testing and QA
This feature was extensively tested in unit tests, testloop tests and in
forknet.

# Checklist
- [ ] Link to nightly nayduck run https://nayduck.nearone.org/#/run/1142
- [x] Update CHANGELOG.md to include this protocol feature in the
`Unreleased` section.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wacban wacban

Labels
Near Core T-core Team: issues relevant to the core team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@wacban @walnut-the-cat