Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance heartbeat processing capabilities #7897

Closed
7 tasks done
nolouch opened this issue Mar 7, 2024 · 0 comments
Closed
7 tasks done

Enhance heartbeat processing capabilities #7897

nolouch opened this issue Mar 7, 2024 · 0 comments
Labels
type/development The issue belongs to a development tasks

Comments

@nolouch
Copy link
Contributor

nolouch commented Mar 7, 2024

Development Task

Sigle node processing capacity

Improve the diagnosis process and implement hierarchical processing for Heartbeat logic

Optimize Lock and Reduce lock holding time

Improve the memory alloc

Scalability

  • [TBD] Proposal
    • only metadata(lookup) part, statics part not consider
@nolouch nolouch added the type/development The issue belongs to a development tasks label Mar 7, 2024
ti-chi-bot bot added a commit that referenced this issue Apr 8, 2024
ref #7897

pkg/schedule: optimize the lock usage of the operator controller
- use sync.Map for operators, which is friendly for check operators on the heartbeat hot path
- reduce the lock hold time by splitting the locks.

Signed-off-by: nolouch <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot bot added a commit that referenced this issue Apr 9, 2024
ref #7897

pkg/ratelimit: Introduce an executor that can run with a limiter
- async runner can run the task with the limiter, the task will parallel run.
- sync runner used to keep the behavior like before

Signed-off-by: nolouch <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot bot added a commit that referenced this issue Apr 11, 2024
ref #7897, close #8049

pkg/schedule: put merge operators together to maintain atomicity

Signed-off-by: nolouch <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot bot pushed a commit that referenced this issue Apr 15, 2024
ref #7897, close #8049

pkg/schedule: put merge operators together to maintain atomicity

Signed-off-by: nolouch <[email protected]>

Co-authored-by: nolouch <[email protected]>
ti-chi-bot bot pushed a commit that referenced this issue Apr 15, 2024
ref #7897

*: Optimize the heartbeat process with the concurrent runner 
- add a switch about the runner
- move some tasks out, not block the heartbeat process

Signed-off-by: nolouch <[email protected]>
ti-chi-bot bot added a commit that referenced this issue Apr 24, 2024
ref #7897

Optimize heartbeat process
- Split the statics updates on the subtree

Signed-off-by: nolouch <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@rleungx rleungx pinned this issue Apr 29, 2024
nolouch added a commit to nolouch/pd that referenced this issue Apr 29, 2024
)

ref tikv#7897

Optimize heartbeat process
- Split the statics updates on the subtree

Signed-off-by: nolouch <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot bot added a commit that referenced this issue Apr 30, 2024
ref #7897

Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot bot added a commit that referenced this issue May 6, 2024
ref #7897, close #8131

Signed-off-by: nolouch <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot bot added a commit that referenced this issue May 7, 2024
ref #7897

Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot bot pushed a commit that referenced this issue May 10, 2024
This was referenced May 10, 2024
ti-chi-bot bot added a commit that referenced this issue May 13, 2024
ref #7897

Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot bot added a commit that referenced this issue May 17, 2024
…#8185)

ref #7897, close #8136

Previously, removing overlapped regions in subtrees required finding overlaps for each subtree individually.
This PR introduces a separate tree specifically for checking overlaps in the subtrees, thereby reducing lookup times and improving performance.

Signed-off-by: JmPotato <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Co-authored-by: Hu# <[email protected]>
ti-chi-bot bot pushed a commit that referenced this issue May 18, 2024
ref #7897

core/region: fix the sub tree may not consist of root tree

Signed-off-by: nolouch <[email protected]>
ti-chi-bot bot pushed a commit that referenced this issue May 21, 2024
…che reset (#8201)

ref #7897

#8132 introduced a count reference to ensure the consistency between
the root tree and subtrees, however, the reference count was not handled correctly during the cache
reset process, which may lead to incorrect reference counts in later heartbeat processing.

Signed-off-by: JmPotato <[email protected]>
ti-chi-bot bot pushed a commit that referenced this issue May 21, 2024
ti-chi-bot bot pushed a commit that referenced this issue May 21, 2024
ti-chi-bot bot pushed a commit that referenced this issue May 22, 2024
ref #7897, close #8207

pkg/statistics: reduce the memory alloc for hot statistics

Signed-off-by: nolouch <[email protected]>
ti-chi-bot bot pushed a commit that referenced this issue May 22, 2024
…8209)

ref #7897

metrics: update grafana template to add heartbeat latency overview

Signed-off-by: nolouch <[email protected]>
ti-chi-bot bot added a commit that referenced this issue May 22, 2024
ref #7897

Optimize the efficiency of random regions selecting.

Signed-off-by: JmPotato <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot bot added a commit that referenced this issue May 23, 2024
ref #7897

metrics: fix the duplicate avg metrics

Signed-off-by: nolouch <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
okJiang pushed a commit to okJiang/pd that referenced this issue May 23, 2024
ref tikv#7897

metrics: fix the duplicate avg metrics

Signed-off-by: nolouch <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: okJiang <[email protected]>
okJiang pushed a commit to okJiang/pd that referenced this issue May 23, 2024
ref tikv#7897

metrics: fix the duplicate avg metrics

Signed-off-by: nolouch <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: okJiang <[email protected]>
ti-chi-bot bot pushed a commit that referenced this issue May 23, 2024
…a single range (#8211)

ref #7897

In the case of only one key range, follow the fast path logic to avoid unnecessary random and loop operations.

Signed-off-by: JmPotato <[email protected]>
ti-chi-bot bot added a commit that referenced this issue May 28, 2024
ref #7897

Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot bot added a commit that referenced this issue May 28, 2024
ref #7897

Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot bot added a commit that referenced this issue Jun 13, 2024
ref #7897

Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot bot added a commit that referenced this issue Jun 13, 2024
ref #7897

Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@nolouch nolouch closed this as completed Aug 1, 2024
@JmPotato JmPotato unpinned this issue Aug 6, 2024
ti-chi-bot bot added a commit that referenced this issue Aug 22, 2024
ref #7897

Signed-off-by: okJiang <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/development The issue belongs to a development tasks
Projects
None yet
Development

No branches or pull requests

1 participant