schedule: support patrol region concurrency #8094

lhy1024 · 2024-04-18T08:29:21Z

What problem does this PR solve?

Issue Number: Close #7963 #7706

What is changed and how does it work?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

Release note

None.

ti-chi-bot · 2024-04-18T08:29:23Z

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

ti-chi-bot · 2024-04-18T08:29:23Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Signed-off-by: lhy1024 <[email protected]>

nolouch

overall lgtm, is this PR ready?

lhy1024 · 2024-05-14T03:32:03Z

overall lgtm, is this PR ready?

I am preparing some tests for different scenarios

Signed-off-by: lhy1024 <[email protected]>

lhy1024 · 2024-05-24T02:24:19Z

pkg/schedule/config/config.go

@@ -63,6 +63,8 @@ const (
 	defaultRegionScoreFormulaVersion = "v2"
 	defaultLeaderSchedulePolicy      = "count"
 	defaultStoreLimitVersion         = "v1"
+	defaultPatrolRegionConcurrency   = 1
+	defaultPatrolRegionBatchLimit    = 128


Maybe we can use max(128,region_count/1024)

pkg/schedule/operator/operator_controller.go

Signed-off-by: lhy1024 <[email protected]>

codecov · 2024-05-24T04:04:30Z

Codecov Report

Attention: Patch coverage is 85.26316% with 14 lines in your changes are missing coverage. Please review.

Project coverage is 77.36%. Comparing base (4cd42b3) to head (ab9ef1e).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8094      +/-   ##
==========================================
+ Coverage   77.29%   77.36%   +0.07%     
==========================================
  Files         471      471              
  Lines       61445    61515      +70     
==========================================
+ Hits        47491    47590      +99     
+ Misses      10395    10362      -33     
- Partials     3559     3563       +4

Flag	Coverage Δ
unittests	`77.36% <85.26%> (+0.07%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Signed-off-by: lhy1024 <[email protected]>

okJiang

rest lgtm

okJiang · 2024-08-29T03:37:03Z

pkg/schedule/checker/checker_controller.go

+			// wait for the regionChan to be drained
+			if len(c.patrolRegionContext.regionChan) > 0 {
+				continue
+			}


Need we wait for? If the regionChan is full, it will be blocked.

okJiang · 2024-08-29T04:14:43Z

pkg/schedule/checker/checker_controller.go

+}
+
+func (p *PatrolRegionContext) stop() {
+	close(p.regionChan)


It is better to add a failpoint to wait for all rest region consumed in regionChan before close regionChan, rather than time.Sleep(100 * time.Millisecond) in L171. This is more stable.

If possible, we can always wait it even if it is not in testing.

time.Sleep(100 * time.Millisecond) in L171 is used to wait that regions to be consumed.

co.PatrolRegions() re.Empty(oc.GetOperators())

For example, if we enable this failpoint, it will wait 100 ms for goroutine consuming regions. And it will check re.Empty(oc.GetOperators()) immediately after failpoint.

I understand the role of L171. But I think sleep is a destabilizing factor. And it's fine to wait for it to finish consuming, or actively consume all regions before exiting and then exit.

remove sleep, PTAL

tools/pd-ctl/tests/config/config_test.go

Signed-off-by: lhy1024 <[email protected]>

bufferflies · 2024-09-24T02:50:06Z

pkg/schedule/checker/checker_controller.go

+		// Stop the old workers and start the new workers.
+		c.patrolRegionContext.workersCancel()
+		c.patrolRegionContext.wg.Wait()
+		c.patrolRegionContext.workersCtx, c.patrolRegionContext.workersCancel = context.WithCancel(c.ctx)


Can we adjust the workers more gracefully? For example, if the new worker count is more than the current workers, we can scale out more wroker and no need to build all workers.

I'm not sure it's necessary, and generally speaking we don't change this configuration very often.

I'm afraid that it maybe wait some time to stop all and start all wokers.

pkg/schedule/checker/checker_controller.go

Signed-off-by: lhy1024 <[email protected]>

ti-chi-bot · 2024-09-25T08:50:26Z

[LGTM Timeline notifier]

Timeline:

2024-06-13 06:42:23.528767754 +0000 UTC m=+619697.582079678: ☑️ agreed by JmPotato.
2024-06-13 06:42:55.992737053 +0000 UTC m=+619730.046048978: ✖️🔁 reset by JmPotato.
2024-09-25 08:50:25.320316515 +0000 UTC m=+1642295.060740453: ☑️ agreed by okJiang.

ti-chi-bot · 2024-09-25T08:50:27Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: okJiang
Once this PR has been reviewed and has the lgtm label, please ask for approval from lhy1024 and additionally assign benmeadowcroft for approval(Please ensuring that each of them provides their approval before proceeding). For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS
pkg/mcs/scheduling/server/config/OWNERS
pkg/schedule/config/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot bot added do-not-merge/needs-linked-issue do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none Denotes a PR that doesn't merit a release note. labels Apr 18, 2024

ti-chi-bot bot requested review from JmPotato and rleungx April 18, 2024 08:29

ti-chi-bot bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Apr 18, 2024

checker: add patrol region concurrency

d1f4b8a

Signed-off-by: lhy1024 <[email protected]>

lhy1024 force-pushed the patrol-concurrency branch from 4c59016 to d1f4b8a Compare April 22, 2024 09:52

lhy1024 added 3 commits April 22, 2024 18:21

speedup drain

6d0fbd4

Signed-off-by: lhy1024 <[email protected]>

fix config

9b43d61

Signed-off-by: lhy1024 <[email protected]>

make config

b2b4f39

Signed-off-by: lhy1024 <[email protected]>

lhy1024 force-pushed the patrol-concurrency branch from 2e61d30 to 948dc77 Compare May 9, 2024 15:33

nolouch reviewed May 14, 2024

View reviewed changes

lhy1024 added 4 commits May 16, 2024 19:54

update

cb285f6

Signed-off-by: lhy1024 <[email protected]>

Merge branch 'master' of github.com:tikv/pd into patrol-concurrency-2

2e14405

Signed-off-by: lhy1024 <[email protected]>

fix race

351ef5c

Signed-off-by: lhy1024 <[email protected]>

Merge branch 'master' of github.com:tikv/pd into patrol-concurrency-2

c198b08

lhy1024 force-pushed the patrol-concurrency branch from 948dc77 to c198b08 Compare May 23, 2024 17:56

lhy1024 changed the title ~~Patrol concurrency~~ schedule: support patrol region concurrency May 23, 2024

ti-chi-bot bot removed the do-not-merge/needs-linked-issue label May 23, 2024

lhy1024 marked this pull request as ready for review May 23, 2024 17:58

ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 23, 2024

lhy1024 commented May 24, 2024

View reviewed changes

pkg/schedule/operator/operator_controller.go Outdated Show resolved Hide resolved

fix test

a0ec33d

Signed-off-by: lhy1024 <[email protected]>

lhy1024 force-pushed the patrol-concurrency branch from e71f635 to a0ec33d Compare May 24, 2024 03:56

lhy1024 added 5 commits July 16, 2024 15:38

refactor

cf01076

Signed-off-by: lhy1024 <[email protected]>

fix test and add metrics

cc51a2e

Signed-off-by: lhy1024 <[email protected]>

fix failpoint

9f7406a

Signed-off-by: lhy1024 <[email protected]>

Merge branch 'master' of github.com:tikv/pd into patrol-concurrency-4

bd4ca79

Signed-off-by: lhy1024 <[email protected]>

fix conflict

ae0778f

Signed-off-by: lhy1024 <[email protected]>

lhy1024 force-pushed the patrol-concurrency branch from ff02183 to ae0778f Compare July 16, 2024 09:08

lhy1024 added 2 commits July 22, 2024 23:41

Merge branch 'master' of github.com:tikv/pd into patrol-concurrency

ecb8d8b

Merge branch 'master' of github.com:tikv/pd into patrol-concurrency

8259ad0

Signed-off-by: lhy1024 <[email protected]>

ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Aug 1, 2024

lhy1024 added 2 commits August 1, 2024 19:01

fix

18db300

Signed-off-by: lhy1024 <[email protected]>

Merge branch 'master' of github.com:tikv/pd into patrol-concurrency

5ead31e

Signed-off-by: lhy1024 <[email protected]>

ti-chi-bot bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Aug 8, 2024

lhy1024 and others added 2 commits August 8, 2024 19:18

fix lint

6bdb436

Signed-off-by: lhy1024 <[email protected]>

Merge branch 'master' into patrol-concurrency

40a2e02

okJiang reviewed Aug 29, 2024

View reviewed changes

address comments

8cd8825

Signed-off-by: lhy1024 <[email protected]>

bufferflies reviewed Sep 24, 2024

View reviewed changes

pkg/schedule/checker/checker_controller.go Show resolved Hide resolved

lhy1024 added 4 commits September 24, 2024 22:00

address comments

bcd5018

Signed-off-by: lhy1024 <[email protected]>

Merge branch 'master' of github.com:tikv/pd into patrol-concurrency

acb4244

avoid potential data race

457da3d

Signed-off-by: lhy1024 <[email protected]>

address comments: remove sleep in failpoint

64abc3c

Signed-off-by: lhy1024 <[email protected]>

ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 25, 2024

okJiang approved these changes Sep 25, 2024

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Sep 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

schedule: support patrol region concurrency #8094

schedule: support patrol region concurrency #8094

lhy1024 commented Apr 18, 2024 •

edited

Loading

ti-chi-bot bot commented Apr 18, 2024

ti-chi-bot bot commented Apr 18, 2024

nolouch left a comment

lhy1024 commented May 14, 2024

lhy1024 May 24, 2024

codecov bot commented May 24, 2024 •

edited

Loading

okJiang left a comment

okJiang Aug 29, 2024

okJiang Aug 29, 2024

lhy1024 Aug 29, 2024

okJiang Aug 30, 2024

lhy1024 Sep 25, 2024

bufferflies Sep 24, 2024

lhy1024 Sep 24, 2024

bufferflies Sep 24, 2024

ti-chi-bot bot commented Sep 25, 2024

ti-chi-bot bot commented Sep 25, 2024

schedule: support patrol region concurrency #8094

Are you sure you want to change the base?

schedule: support patrol region concurrency #8094

Conversation

lhy1024 commented Apr 18, 2024 • edited Loading

What problem does this PR solve?

What is changed and how does it work?

Check List

Release note

ti-chi-bot bot commented Apr 18, 2024

ti-chi-bot bot commented Apr 18, 2024

nolouch left a comment

Choose a reason for hiding this comment

lhy1024 commented May 14, 2024

Choose a reason for hiding this comment

codecov bot commented May 24, 2024 • edited Loading

Codecov Report

okJiang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Sep 25, 2024

[LGTM Timeline notifier]

ti-chi-bot bot commented Sep 25, 2024

lhy1024 commented Apr 18, 2024 •

edited

Loading

codecov bot commented May 24, 2024 •

edited

Loading