Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dm-relay: optimize cpu usage when enabling relay #3469

Merged
merged 30 commits into from
Nov 22, 2021
Merged

dm-relay: optimize cpu usage when enabling relay #3469

merged 30 commits into from
Nov 22, 2021

Conversation

D3Hunter
Copy link
Contributor

@D3Hunter D3Hunter commented Nov 16, 2021 •
edited
Loading

What problem does this PR solve?

after subtask 2 and 3 here solved, latency after enabling relay do meet requirement, but cpu usage is high. this pr optimize cpu usage when enabling relay.

What is changed and how it works?

  • open binlog file once and read first event once in relay reader
  • check active file and offset when waiting for new events to make more accurate decision
  • reduce memory allocation to reduce GC load
    • reuse buffer when reading binlog, see go-mysql 644
    • reuse/remove unnecessary timer
    • log debug events as needed in relay
  • others, such as parse uuid as needed

Check List

Tests

  • Unit test

Release note 

reduce CPU usage when enabling relay

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Nov 16, 2021 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • Ehco1996
  • GMHDBJD

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. contribution This PR is from a community contributor. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 16, 2021
@lichunzhu
Copy link
Contributor

/area dm

@ti-chi-bot ti-chi-bot added the area/dm Issues or PRs related to DM. label Nov 16, 2021
@D3Hunter
Copy link
Contributor Author

/area dm

@D3Hunter
Copy link
Contributor Author

/run-all-tests

@codecov-commenter
Copy link

Codecov Report

Merging #3469 (b7221d6) into master (24030f8) will increase coverage by 0.8209%.
The diff coverage is 64.0186%.

@@               Coverage Diff                @@
##             master      #3469        +/-   ##
================================================
+ Coverage   56.5356%   57.3565%   +0.8209%     
================================================
  Files           211        228        +17     
  Lines         22798      23462       +664     
================================================
+ Hits          12889      13457       +568     
- Misses         8598       8643        +45     
- Partials       1311       1362        +51

@D3Hunter D3Hunter marked this pull request as ready for review November 16, 2021 08:03
@ti-chi-bot ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 16, 2021
@ti-chi-bot ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 22, 2021
@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Nov 22, 2021
GMHDBJD
GMHDBJD approved these changes Nov 22, 2021
View reviewed changes
Copy link
Contributor

@GMHDBJD GMHDBJD left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Nov 22, 2021
lance6716
lance6716 reviewed Nov 22, 2021
View reviewed changes
Copy link
Contributor

@lance6716 lance6716 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/hold

will finish review soon

dm/relay/upstream_reader.go Show resolved Hide resolved
dm/relay/upstream_reader.go Show resolved Hide resolved
dm/relay/streamer.go Show resolved Hide resolved
@ti-chi-bot ti-chi-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 22, 2021
lance6716
lance6716 reviewed Nov 22, 2021
View reviewed changes
Copy link
Contributor

@lance6716 lance6716 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest lgtm

dm/relay/local_reader.go Outdated Show resolved Hide resolved
dm/relay/local_reader.go Show resolved Hide resolved
lichunzhu
lichunzhu approved these changes Nov 22, 2021
View reviewed changes
Copy link
Contributor

@lichunzhu lichunzhu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, great job

@lance6716
Copy link
Contributor

/unhold

@ti-chi-bot ti-chi-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 22, 2021
@D3Hunter
Copy link
Contributor Author

/run-all-tests

@lance6716
Copy link
Contributor

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 84c5ae9

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Nov 22, 2021
@ti-chi-bot
Copy link
Member

@D3Hunter: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot ti-chi-bot merged commit 0076d09 into pingcap:master Nov 22, 2021
@D3Hunter D3Hunter deleted the opt-relay-cpu branch November 23, 2021 02:10
okJiang pushed a commit to okJiang/tiflow that referenced this pull request Dec 8, 2021
@D3Hunter @okJiang
dm-relay: optimize cpu usage when enabling relay (pingcap#3469)
05f877a
@D3Hunter D3Hunter added this to the v5.4.0 milestone Jan 13, 2022
@niubell niubell mentioned this pull request Jan 14, 2022
Merged
14 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lonng lonng lonng left review comments

@lance6716 lance6716 lance6716 left review comments

@lichunzhu lichunzhu lichunzhu approved these changes

@Ehco1996 Ehco1996 Ehco1996 approved these changes

@GMHDBJD GMHDBJD GMHDBJD approved these changes

Assignees
No one assigned
Labels
area/dm Issues or PRs related to DM. contribution This PR is from a community contributor. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
v5.4.0
Development

Successfully merging this pull request may close these issues.
8 participants
@D3Hunter @ti-chi-bot @lichunzhu @codecov-commenter @lance6716 @lonng @Ehco1996 @GMHDBJD