*: batch process peer task #8213

rleungx · 2024-05-23T10:44:36Z

What problem does this PR solve?

Issue Number: Ref #7897.

What is changed and how does it work?

$ benchstat old.txt new.txt
goos: linux
goarch: amd64
pkg: github.com/tikv/pd/server/cluster
cpu: Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz
                    │    old.txt    │               new.txt                │
                    │    sec/op     │    sec/op     vs base                │
HandleStatsAsync-72   610.2n ± 171%   370.1n ± 27%  -39.35% (p=0.000 n=10)

                    │   old.txt   │              new.txt               │
                    │    B/op     │    B/op     vs base                │
HandleStatsAsync-72   128.00 ± 3%   24.00 ± 0%  -81.25% (p=0.000 n=10)

                    │  old.txt   │              new.txt               │
                    │ allocs/op  │ allocs/op   vs base                │
HandleStatsAsync-72   5.000 ± 0%   3.000 ± 0%  -40.00% (p=0.000 n=10)

Check List

Tests

Unit test

Release note

None.

ti-chi-bot · 2024-05-23T10:44:39Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

JmPotato
lhy1024

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

ti-chi-bot · 2024-05-23T10:44:39Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Signed-off-by: Ryan Leung <[email protected]>

codecov · 2024-05-24T11:03:51Z

Codecov Report

Attention: Patch coverage is 98.41270% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 77.28%. Comparing base (beb91c1) to head (8b3475e).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8213      +/-   ##
==========================================
- Coverage   77.38%   77.28%   -0.11%     
==========================================
  Files         471      471              
  Lines       61478    61457      -21     
==========================================
- Hits        47574    47496      -78     
- Misses      10345    10401      +56     
- Partials     3559     3560       +1

Flag	Coverage Δ
unittests	`77.28% <98.41%> (-0.11%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Signed-off-by: Ryan Leung <[email protected]>

pkg/statistics/hot_peer_cache.go

Signed-off-by: Ryan Leung <[email protected]>

lhy1024

the rest LGTM

lhy1024 · 2024-05-28T04:39:43Z

pkg/statistics/hot_cache.go

@@ -172,14 +173,14 @@ func (w *HotCache) Update(item *HotPeerStat, kind utils.RWType) {

 // CheckWritePeerSync checks the write status, returns update items.
 // This is used for mockcluster, for test purpose.
-func (w *HotCache) CheckWritePeerSync(peer *core.PeerInfo, region *core.RegionInfo) *HotPeerStat {
-	return w.writeCache.checkPeerFlow(peer, region)
+func (w *HotCache) CheckWritePeerSync(region *core.RegionInfo, peers []*metapb.Peer, loads []float64, interval uint64) []*HotPeerStat {


CheckWritePeersSync or CheckWriteRegionSync?

IMO, peer is better

lhy1024 · 2024-05-28T04:40:00Z

pkg/statistics/hot_cache.go

 }

 // CheckReadPeerSync checks the read status, returns update items.
 // This is used for mockcluster, for test purpose.
-func (w *HotCache) CheckReadPeerSync(peer *core.PeerInfo, region *core.RegionInfo) *HotPeerStat {
-	return w.readCache.checkPeerFlow(peer, region)
+func (w *HotCache) CheckReadPeerSync(region *core.RegionInfo, peers []*metapb.Peer, loads []float64, interval uint64) []*HotPeerStat {


rleungx · 2024-05-28T05:31:03Z

/merge

ti-chi-bot · 2024-05-28T05:31:06Z

@rleungx: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2024-05-28T05:31:08Z

This pull request has been accepted and is ready to merge.

Commit hash: d030133

ti-chi-bot · 2024-05-28T05:31:21Z

@rleungx: Your PR was out of date, I have automatically updated it for you.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot bot added do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels May 23, 2024

ti-chi-bot bot requested review from JmPotato and nolouch May 23, 2024 10:44

ti-chi-bot bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label May 23, 2024

rleungx removed request for JmPotato and nolouch May 23, 2024 10:44

fix

52e1681

Signed-off-by: Ryan Leung <[email protected]>

rleungx force-pushed the s branch from b453646 to 52e1681 Compare May 24, 2024 07:32

batch process the peer task

9d88e55

Signed-off-by: Ryan Leung <[email protected]>

rleungx force-pushed the s branch from 8413c4e to 9d88e55 Compare May 24, 2024 10:48

rleungx marked this pull request as ready for review May 24, 2024 10:55

ti-chi-bot bot removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. do-not-merge/needs-linked-issue labels May 24, 2024

rleungx requested review from JmPotato and nolouch May 27, 2024 02:36

rleungx added 3 commits May 27, 2024 14:50

use pool

b89a6eb

Signed-off-by: Ryan Leung <[email protected]>

remove pool

6bf5e38

Signed-off-by: Ryan Leung <[email protected]>

fix

7a65fc7

Signed-off-by: Ryan Leung <[email protected]>

JmPotato reviewed May 27, 2024

View reviewed changes

pkg/statistics/hot_peer_cache.go Outdated Show resolved Hide resolved

pkg/statistics/hot_peer_cache.go Outdated Show resolved Hide resolved

rleungx added 2 commits May 28, 2024 10:30

address comment

09c6b92

Signed-off-by: Ryan Leung <[email protected]>

address comment

d030133

Signed-off-by: Ryan Leung <[email protected]>

JmPotato approved these changes May 28, 2024

View reviewed changes

ti-chi-bot bot added the status/LGT1 Indicates that a PR has LGTM 1. label May 28, 2024

lhy1024 reviewed May 28, 2024

View reviewed changes

lhy1024 approved these changes May 28, 2024

View reviewed changes

ti-chi-bot bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels May 28, 2024

ti-chi-bot bot added the status/can-merge Indicates a PR has been approved by a committer. label May 28, 2024

Merge branch 'master' into s

8b3475e

ti-chi-bot bot merged commit 9d580d0 into tikv:master May 28, 2024
19 checks passed

rleungx deleted the s branch May 28, 2024 05:47

nolouch mentioned this pull request Jun 5, 2024

Enhance heartbeat processing capabilities #7897

Closed

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: batch process peer task #8213

*: batch process peer task #8213

rleungx commented May 23, 2024 •

edited

Loading

ti-chi-bot bot commented May 23, 2024 •

edited

Loading

ti-chi-bot bot commented May 23, 2024

codecov bot commented May 24, 2024 •

edited

Loading

lhy1024 left a comment

lhy1024 May 28, 2024

rleungx May 28, 2024

lhy1024 May 28, 2024

rleungx commented May 28, 2024

ti-chi-bot bot commented May 28, 2024

ti-chi-bot bot commented May 28, 2024

ti-chi-bot bot commented May 28, 2024

*: batch process peer task #8213

*: batch process peer task #8213

Conversation

rleungx commented May 23, 2024 • edited Loading

What problem does this PR solve?

What is changed and how does it work?

Check List

Release note

ti-chi-bot bot commented May 23, 2024 • edited Loading

ti-chi-bot bot commented May 23, 2024

codecov bot commented May 24, 2024 • edited Loading

Codecov Report

lhy1024 left a comment

Choose a reason for hiding this comment

lhy1024 May 28, 2024

Choose a reason for hiding this comment

rleungx May 28, 2024

Choose a reason for hiding this comment

lhy1024 May 28, 2024

Choose a reason for hiding this comment

rleungx commented May 28, 2024

ti-chi-bot bot commented May 28, 2024

ti-chi-bot bot commented May 28, 2024

ti-chi-bot bot commented May 28, 2024

rleungx commented May 23, 2024 •

edited

Loading

ti-chi-bot bot commented May 23, 2024 •

edited

Loading

codecov bot commented May 24, 2024 •

edited

Loading