[to #48] try to fix scheduler problem #90
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: zeminzhou <[email protected]>
|
TODO: Unit Test. |
zeminzhou
changed the title
added 4 commits
April 21, 2022 11:57
Signed-off-by: zeminzhou <[email protected]>
Signed-off-by: zeminzhou <[email protected]>
Signed-off-by: zeminzhou <[email protected]>
Signed-off-by: zeminzhou <[email protected]>
Codecov Report
@@ Coverage Diff @@
## main #90 +/- ##
================================================
+ Coverage 53.4580% 61.2819% +7.8239%
================================================
Files 197 180 -17
Lines 19838 14275 -5563
================================================
- Hits 10605 8748 -1857
+ Misses 8265 4910 -3355
+ Partials 968 617 -351
Flags with carried forward coverage won't be shown. Click here to find out more.
|
pingyu
reviewed
May 6, 2022
| @@ -113,6 +118,27 @@ func (s *oldScheduler) Tick( | |||
| return shouldUpdateState, nil | |||
| } | |||
|
|
|||
| func (s *oldScheduler) diffCurrentKeySpans(currentKeySpans map[model.KeySpanID]regionspan.Span) (map[model.KeySpanID]struct{}, []model.KeySpanID) { | |||
There was a problem hiding this comment.
Please add unit test for this method.
| @@ -246,26 +272,41 @@ func (s *oldScheduler) dispatchToTargetCaptures(pendingJobs []*schedulerJob) { | |||
|
|
|||
| // syncKeySpansWithCurrentKeySpans iterates all current keyspans to check whether it should be listened or not. | |||
| // this function will return schedulerJob to make sure all keyspans will be listened. | |||
| func (s *oldScheduler) syncKeySpansWithCurrentKeySpans() ([]*schedulerJob, error) { | |||
| func (s *oldScheduler) syncKeySpansWithCurrentKeySpans(newKeySpans map[model.KeySpanID]struct{}, needRemoveKeySpans []model.KeySpanID) ([]*schedulerJob, error) { | |||
There was a problem hiding this comment.
Suggest to add unit test for this method.
| @@ -597,6 +601,18 @@ func (p *processor) handleKeySpanOperation(ctx cdcContext.Context) error { | |||
| return nil | |||
| } | |||
|
|
|||
| func (p *processor) checkRelatedKeyspans(relatedKeySpans []model.KeySpanLocation) bool { | |||
There was a problem hiding this comment.
Please add unit test for this method.
Signed-off-by: zeminzhou <[email protected]>
Signed-off-by: zeminzhou <[email protected]>
Signed-off-by: zeminzhou <[email protected]>
Signed-off-by: zeminzhou <[email protected]>
Signed-off-by: zeminzhou <[email protected]>
Signed-off-by: zeminzhou <[email protected]>
haojinming
reviewed
Jun 14, 2022
cdc/cdc/model/owner.go
Outdated
| @@ -193,7 +195,7 @@ type TaskWorkload map[KeySpanID]WorkloadInfo | |||
|
|
|||
| // WorkloadInfo records the workload info of a keyspan | |||
| type WorkloadInfo struct { | |||
| Workload uint64 `json:"workload"` | |||
| Workload int64 `json:"workload"` | |||
There was a problem hiding this comment.
Why change to int64? Can it be negative?
Signed-off-by: zeminzhou <[email protected]>
Signed-off-by: zeminzhou <[email protected]>
haojinming
reviewed
Jun 15, 2022
cdc/pkg/scheduler/workload.go
Outdated
| @@ -88,7 +88,7 @@ func (w workloads) Skewness() float64 { | |||
| } | |||
|
|
|||
| func (w workloads) SelectIdleCapture() model.CaptureID { | |||
| minWorkload := uint64(math.MaxUint64) | |||
| minWorkload := uint64(math.MaxInt64) | |||
There was a problem hiding this comment.
maybe revert this change?
Signed-off-by: zeminzhou <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: zeminzhou [email protected]
What problem does this PR solve?
Keyspans to add & keyspans to remove, must be overlapped. So to be correct, I think we must finish remove jobs first, and then dispatch the add jobs.here
Issue Number: to #48
Problem Description:
Keyspans to add & keyspans to remove, must be overlapped. So to be correct, I think we must finish remove jobs first, and then dispatch the add jobs
What is changed and how does it work?
After each acquisition of the current keyspan, compare the current keyspan with the keyspan of the previous period to establish a corresponding relationship newKeySpan -> needRemoveKeySpan. The running and deployment of the newKeySpan task needs to wait for all the needRemoveKeySpan tasks to stop. This check will be done in Processor.
Code changes