scheduling: sync schedulers from the API server #7076

JmPotato · 2023-09-11T10:15:40Z

What problem does this PR solve?

Issue Number: ref #5839. Should merge after #7082.

What is changed and how does it work?

- Sync the schedulers from the API server.
- Dynamically reload the scheduler config.

Check List

Tests

Unit test
Integration test

Release note

None.

ti-chi-bot · 2023-09-11T10:15:42Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

lhy1024
rleungx

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

codecov · 2023-09-12T02:35:30Z

Codecov Report

Merging #7076 (0dfaa08) into master (1a0c8fb) will decrease coverage by 0.02%.
The diff coverage is 60.45%.

❗ Current head 0dfaa08 differs from pull request most recent head b40721d. Consider uploading reports for the commit b40721d to get more accurate results

@@            Coverage Diff             @@
##           master    #7076      +/-   ##
==========================================
- Coverage   74.43%   74.41%   -0.02%     
==========================================
  Files         437      437              
  Lines       46618    46712      +94     
==========================================
+ Hits        34699    34760      +61     
- Misses       8870     8905      +35     
+ Partials     3049     3047       -2

Flag	Coverage Δ
unittests	`74.41% <60.45%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

pkg/mcs/scheduling/server/cluster.go

pkg/schedule/coordinator.go

tests/integrations/mcs/scheduling/server_test.go

pkg/mcs/scheduling/server/cluster.go

lhy1024 · 2023-09-13T16:43:16Z

pkg/schedule/schedulers/grant_leader.go

@@ -178,6 +178,24 @@ func (s *grantLeaderScheduler) EncodeConfig() ([]byte, error) {
 	return EncodeConfig(s.conf)
 }

+func (s *grantLeaderScheduler) ReloadConfig() error {


Can we reduce the duplication in ReloadConfig?

Different config structures need to have different implementation, I think it's OK to have some duplication here.

pkg/schedule/coordinator.go

JmPotato · 2023-09-14T03:40:40Z

/merge

ti-chi-bot · 2023-09-14T03:40:42Z

@JmPotato: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2023-09-14T03:40:43Z

This pull request has been accepted and is ready to merge.

Commit hash: 0dfaa08

Signed-off-by: JmPotato <[email protected]>

JmPotato · 2023-09-14T03:43:54Z

/merge

ti-chi-bot · 2023-09-14T03:43:56Z

@JmPotato: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2023-09-14T03:43:57Z

This pull request has been accepted and is ready to merge.

Commit hash: 26d7e18

lhy1024

TestSchedulerSync is failed

Signed-off-by: JmPotato <[email protected]>

JmPotato · 2023-09-14T06:34:17Z

/merge

ti-chi-bot · 2023-09-14T06:34:20Z

@JmPotato: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2023-09-14T06:34:21Z

This pull request has been accepted and is ready to merge.

Commit hash: b40721d

ref tikv#5839 - Sync the schedulers from the API server. - Dynamically reload the scheduler config. Signed-off-by: JmPotato <[email protected]>

JmPotato added component/schedule Scheduling logic. component/mcs Microservice. labels Sep 11, 2023

JmPotato requested review from lhy1024 and rleungx September 11, 2023 10:15

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 11, 2023

rleungx reviewed Sep 12, 2023

View reviewed changes

pkg/mcs/scheduling/server/cluster.go Outdated Show resolved Hide resolved

pkg/schedule/coordinator.go Show resolved Hide resolved

rleungx reviewed Sep 12, 2023

View reviewed changes

tests/integrations/mcs/scheduling/server_test.go Outdated Show resolved Hide resolved

JmPotato mentioned this pull request Sep 12, 2023

mcs/scheduling: register the scheduler handler in API service mode #7082

Merged

JmPotato force-pushed the scheduling_sync_scheduler branch from 02f8a29 to 0f64a55 Compare September 13, 2023 08:40

ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 13, 2023

JmPotato requested a review from rleungx September 13, 2023 08:45

rleungx reviewed Sep 13, 2023

View reviewed changes

pkg/mcs/scheduling/server/cluster.go Outdated Show resolved Hide resolved

lhy1024 reviewed Sep 13, 2023

View reviewed changes

JmPotato force-pushed the scheduling_sync_scheduler branch from 540500d to 0dfaa08 Compare September 14, 2023 01:54

JmPotato requested review from lhy1024 and rleungx September 14, 2023 01:54

rleungx approved these changes Sep 14, 2023

View reviewed changes

ti-chi-bot bot added the status/LGT1 Indicates that a PR has LGTM 1. label Sep 14, 2023

lhy1024 approved these changes Sep 14, 2023

View reviewed changes

ti-chi-bot bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Sep 14, 2023

ti-chi-bot bot added the status/can-merge Indicates a PR has been approved by a committer. label Sep 14, 2023

Sync schedulers from the API server and load the config dynamically

289e30e

Signed-off-by: JmPotato <[email protected]>

JmPotato added 3 commits September 14, 2023 11:43

Refine the comments

40d9b6d

Signed-off-by: JmPotato <[email protected]>

Refine UpdateScheduler

33b4bcb

Signed-off-by: JmPotato <[email protected]>

Address the comment

26d7e18

Signed-off-by: JmPotato <[email protected]>

JmPotato force-pushed the scheduling_sync_scheduler branch from 0dfaa08 to 26d7e18 Compare September 14, 2023 03:43

ti-chi-bot bot removed the status/can-merge Indicates a PR has been approved by a committer. label Sep 14, 2023

ti-chi-bot bot added the status/can-merge Indicates a PR has been approved by a committer. label Sep 14, 2023

lhy1024 reviewed Sep 14, 2023

View reviewed changes

Fix the test

b40721d

Signed-off-by: JmPotato <[email protected]>

ti-chi-bot bot removed the status/can-merge Indicates a PR has been approved by a committer. label Sep 14, 2023

ti-chi-bot bot added the status/can-merge Indicates a PR has been approved by a committer. label Sep 14, 2023

ti-chi-bot bot merged commit e295e62 into tikv:master Sep 14, 2023
19 checks passed

JmPotato deleted the scheduling_sync_scheduler branch September 14, 2023 07:02

rleungx mentioned this pull request Sep 15, 2023

Separate scheduling into an independent service #5839

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

scheduling: sync schedulers from the API server #7076

scheduling: sync schedulers from the API server #7076

JmPotato commented Sep 11, 2023 •

edited

Loading

ti-chi-bot bot commented Sep 11, 2023 •

edited

Loading

codecov bot commented Sep 12, 2023 •

edited

Loading

lhy1024 Sep 13, 2023

JmPotato Sep 14, 2023

JmPotato commented Sep 14, 2023

ti-chi-bot bot commented Sep 14, 2023

ti-chi-bot bot commented Sep 14, 2023

JmPotato commented Sep 14, 2023

ti-chi-bot bot commented Sep 14, 2023

ti-chi-bot bot commented Sep 14, 2023

lhy1024 left a comment

JmPotato commented Sep 14, 2023

ti-chi-bot bot commented Sep 14, 2023

ti-chi-bot bot commented Sep 14, 2023

scheduling: sync schedulers from the API server #7076

scheduling: sync schedulers from the API server #7076

Conversation

JmPotato commented Sep 11, 2023 • edited Loading

What problem does this PR solve?

What is changed and how does it work?

Check List

Release note

ti-chi-bot bot commented Sep 11, 2023 • edited Loading

codecov bot commented Sep 12, 2023 • edited Loading

Codecov Report

lhy1024 Sep 13, 2023

Choose a reason for hiding this comment

JmPotato Sep 14, 2023

Choose a reason for hiding this comment

JmPotato commented Sep 14, 2023

ti-chi-bot bot commented Sep 14, 2023

ti-chi-bot bot commented Sep 14, 2023

JmPotato commented Sep 14, 2023

ti-chi-bot bot commented Sep 14, 2023

ti-chi-bot bot commented Sep 14, 2023

lhy1024 left a comment

Choose a reason for hiding this comment

JmPotato commented Sep 14, 2023

ti-chi-bot bot commented Sep 14, 2023

ti-chi-bot bot commented Sep 14, 2023

JmPotato commented Sep 11, 2023 •

edited

Loading

ti-chi-bot bot commented Sep 11, 2023 •

edited

Loading

codecov bot commented Sep 12, 2023 •

edited

Loading