cluster: dynamic progress time window for offline scene #7722

CabinfeverB · 2024-01-17T03:10:35Z

What problem does this PR solve?

Issue Number: Close #7726

What is changed and how does it work?

time window length can be changed
Two Pointers algorithm is used to maintain statistics in the time window to avoid data loss when patrol cost time changes

Check List

Tests

Unit test
Manual test (add detailed scripts or steps below)

tiup playground nightly --tiflash 2 --pd.binpath=/Users/jiangyongbo/github/pd/bin/pd-server
sysbench --mysql-host=127.0.0.1 --mysql-user=root --mysql-port=4000 --mysql-db=sbtest --percentile=99 --report-interval=1 --tables=32 --table_size=500000 oltp_read_only --range_selects --threads=4 --time=9800 prepare
alter table sbtest.sbtset1 set tiflash replica 1;
alter table sbtest.sbtset9 set tiflash replica 1;
alter table sbtest.sbtset17 set tiflash replica 1;
alter table sbtest.sbtset25 set tiflash replica 1;
config set patrol-region-interval 22s

Code changes

Has the configuration change
Has HTTP API interfaces changed (Don't forget to add the declarative for the new API)
Has persistent data change

Side effects

Possible performance regression
Increased code complexity
Breaking backward compatibility

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
PR to update pingcap/tiup:
Need to cherry-pick to the release branch

Release note

None.

Signed-off-by: Cabinfever_B <[email protected]>

ti-chi-bot · 2024-01-17T03:10:37Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

HuSharp
rleungx

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

ti-chi-bot · 2024-01-17T03:10:38Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Signed-off-by: Cabinfever_B <[email protected]>

codecov · 2024-01-17T08:19:36Z

Codecov Report

Merging #7722 (d2ff3ce) into master (a44b208) will decrease coverage by 0.07%.
Report is 1 commits behind head on master.
The diff coverage is 98.00%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7722      +/-   ##
==========================================
- Coverage   73.66%   73.59%   -0.07%     
==========================================
  Files         436      436              
  Lines       48474    48503      +29     
==========================================
- Hits        35707    35696      -11     
- Misses       9721     9742      +21     
- Partials     3046     3065      +19

Flag	Coverage Δ
unittests	`73.59% <98.00%> (-0.07%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

pkg/schedule/coordinator.go

Signed-off-by: Cabinfever_B <[email protected]>

pkg/progress/progress.go

Signed-off-by: Cabinfever_B <[email protected]>

pkg/progress/progress.go

Signed-off-by: Cabinfever_B <[email protected]>

pkg/progress/progress_test.go

Signed-off-by: Cabinfever_B <[email protected]>

pkg/schedule/coordinator.go

Signed-off-by: Cabinfever_B <[email protected]>

HuSharp

LGTM!

CabinfeverB · 2024-04-01T02:23:06Z

/merge

ti-chi-bot · 2024-04-01T02:23:08Z

@CabinfeverB: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2024-04-01T02:23:10Z

This pull request has been accepted and is ready to merge.

Commit hash: 7e3cf28

close tikv#7726 Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot · 2024-04-01T02:33:53Z

In response to a cherrypick label: new pull request created to branch release-7.5: #8005.

ti-chi-bot · 2024-04-01T02:34:29Z

In response to a cherrypick label: new pull request created to branch release-6.5: #8006.

close tikv#7726 Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot · 2024-04-01T02:35:06Z

In response to a cherrypick label: new pull request created to branch release-7.1: #8007.

close tikv#7726 Signed-off-by: ti-chi-bot <[email protected]>

close #7726 Signed-off-by: ti-chi-bot <[email protected]> Signed-off-by: Cabinfever_B <[email protected]> Co-authored-by: Yongbo Jiang <[email protected]> Co-authored-by: Cabinfever_B <[email protected]>

close #7726 Signed-off-by: ti-chi-bot <[email protected]> Signed-off-by: Cabinfever_B <[email protected]> Co-authored-by: Yongbo Jiang <[email protected]> Co-authored-by: Cabinfever_B <[email protected]> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

dynamic progress time window

2974a3d

Signed-off-by: Cabinfever_B <[email protected]>

CabinfeverB requested a review from rleungx January 17, 2024 03:10

ti-chi-bot bot added do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. labels Jan 17, 2024

ti-chi-bot bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jan 17, 2024

dynamic progress time window

1740480

Signed-off-by: Cabinfever_B <[email protected]>

CabinfeverB marked this pull request as ready for review January 17, 2024 07:58

ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 17, 2024

dynamic progress time window

0fb44f9

Signed-off-by: Cabinfever_B <[email protected]>

rleungx reviewed Jan 17, 2024

View reviewed changes

pkg/schedule/coordinator.go Outdated Show resolved Hide resolved

CabinfeverB added 2 commits January 17, 2024 17:23

address comment

d572853

Signed-off-by: Cabinfever_B <[email protected]>

address comment

32b7aa6

Signed-off-by: Cabinfever_B <[email protected]>

CabinfeverB requested review from JmPotato, lhy1024 and rleungx January 31, 2024 15:18

CabinfeverB added 3 commits February 1, 2024 14:53

Merge branch 'master' into offline_time_window

764f0bc

address comment

1a49ec9

Signed-off-by: Cabinfever_B <[email protected]>

address comment

e2e3641

Signed-off-by: Cabinfever_B <[email protected]>

ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Feb 2, 2024

merge master

6f87cb1

Signed-off-by: Cabinfever_B <[email protected]>

rleungx reviewed Mar 28, 2024

View reviewed changes

pkg/progress/progress.go Outdated Show resolved Hide resolved

address comment

9cf28ee

Signed-off-by: Cabinfever_B <[email protected]>

HuSharp reviewed Mar 29, 2024

View reviewed changes

pkg/progress/progress.go Outdated Show resolved Hide resolved

pkg/progress/progress.go Outdated Show resolved Hide resolved

pkg/progress/progress.go Outdated Show resolved Hide resolved

address comment

5cccb7e

Signed-off-by: Cabinfever_B <[email protected]>

HuSharp reviewed Mar 29, 2024

View reviewed changes

pkg/progress/progress_test.go Outdated Show resolved Hide resolved

CabinfeverB added 2 commits March 29, 2024 17:48

address comment

f67bdaf

Signed-off-by: Cabinfever_B <[email protected]>

address comment

8a8838f

Signed-off-by: Cabinfever_B <[email protected]>

HuSharp reviewed Mar 29, 2024

View reviewed changes

pkg/schedule/coordinator.go Outdated Show resolved Hide resolved

address comment

7e3cf28

Signed-off-by: Cabinfever_B <[email protected]>

HuSharp approved these changes Apr 1, 2024

View reviewed changes

ti-chi-bot bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Apr 1, 2024

ti-chi-bot bot added the status/can-merge Indicates a PR has been approved by a committer. label Apr 1, 2024

Merge branch 'master' into offline_time_window

d2ff3ce

ti-chi-bot bot merged commit 945e29c into tikv:master Apr 1, 2024
22 checks passed

ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this pull request Apr 1, 2024

This is an automated cherry-pick of tikv#7722

dc62563

close tikv#7726 Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Apr 1, 2024

cluster: dynamic progress time window for offline scene (#7722) #8005

Merged

ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this pull request Apr 1, 2024

This is an automated cherry-pick of tikv#7722

7c167f4

close tikv#7726 Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Apr 1, 2024

cluster: dynamic progress time window for offline scene (#7722) #8006

Merged

ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this pull request Apr 1, 2024

This is an automated cherry-pick of tikv#7722

65f0ea2

close tikv#7726 Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Apr 1, 2024

cluster: dynamic progress time window for offline scene (#7722) #8007

Merged

rleungx mentioned this pull request Apr 8, 2024

Data race when updating store progress #8036

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cluster: dynamic progress time window for offline scene #7722

cluster: dynamic progress time window for offline scene #7722

CabinfeverB commented Jan 17, 2024 •

edited

Loading

ti-chi-bot bot commented Jan 17, 2024 •

edited

Loading

ti-chi-bot bot commented Jan 17, 2024

codecov bot commented Jan 17, 2024 •

edited

Loading

HuSharp left a comment

CabinfeverB commented Apr 1, 2024

ti-chi-bot bot commented Apr 1, 2024

ti-chi-bot bot commented Apr 1, 2024

ti-chi-bot commented Apr 1, 2024

ti-chi-bot commented Apr 1, 2024

ti-chi-bot commented Apr 1, 2024

cluster: dynamic progress time window for offline scene #7722

cluster: dynamic progress time window for offline scene #7722

Conversation

CabinfeverB commented Jan 17, 2024 • edited Loading

What problem does this PR solve?

What is changed and how does it work?

Check List

Release note

ti-chi-bot bot commented Jan 17, 2024 • edited Loading

ti-chi-bot bot commented Jan 17, 2024

codecov bot commented Jan 17, 2024 • edited Loading

Codecov Report

HuSharp left a comment

Choose a reason for hiding this comment

CabinfeverB commented Apr 1, 2024

ti-chi-bot bot commented Apr 1, 2024

ti-chi-bot bot commented Apr 1, 2024

ti-chi-bot commented Apr 1, 2024

ti-chi-bot commented Apr 1, 2024

ti-chi-bot commented Apr 1, 2024

CabinfeverB commented Jan 17, 2024 •

edited

Loading

ti-chi-bot bot commented Jan 17, 2024 •

edited

Loading

codecov bot commented Jan 17, 2024 •

edited

Loading