dr-autosync: add metrics #7110

disksing · 2023-09-19T07:48:28Z

What problem does this PR solve?

Issue Number: Close #6975

What is changed and how does it work?

add some dr-autosync related metrics

numbers are used to distinguish different states, the map is

1: sync
2: async_wait
3: async
4: sync_recover

Check List

Tests

Manual test (add detailed scripts or steps below)

sync state

# HELP pd_replication_dr_recover_progress Progress of sync_recover process
# TYPE pd_replication_dr_recover_progress gauge
pd_replication_dr_recover_progress 0
# HELP pd_replication_dr_recovered_region Number of recovered regions
# TYPE pd_replication_dr_recovered_region gauge
pd_replication_dr_recovered_region 0
# HELP pd_replication_dr_state State of DR
# TYPE pd_replication_dr_state gauge
pd_replication_dr_state 1
# HELP pd_replication_dr_state_id State ID of DR
# TYPE pd_replication_dr_state_id gauge
pd_replication_dr_state_id 10
# HELP pd_replication_dr_tick_total Counter of background state check count
# TYPE pd_replication_dr_tick_total counter
pd_replication_dr_tick_total 64

async wait

# HELP pd_replication_dr_recover_progress Progress of sync_recover process
# TYPE pd_replication_dr_recover_progress gauge
pd_replication_dr_recover_progress 0
# HELP pd_replication_dr_recovered_region Number of recovered regions
# TYPE pd_replication_dr_recovered_region gauge
pd_replication_dr_recovered_region 0
# HELP pd_replication_dr_state State of DR
# TYPE pd_replication_dr_state gauge
pd_replication_dr_state 2
# HELP pd_replication_dr_state_id State ID of DR
# TYPE pd_replication_dr_state_id gauge
pd_replication_dr_state_id 221
# HELP pd_replication_dr_tick_total Counter of background state check count
# TYPE pd_replication_dr_tick_total counter
pd_replication_dr_tick_total 311

async

# HELP pd_replication_dr_recover_progress Progress of sync_recover process
# TYPE pd_replication_dr_recover_progress gauge
pd_replication_dr_recover_progress 0
# HELP pd_replication_dr_recovered_region Number of recovered regions
# TYPE pd_replication_dr_recovered_region gauge
pd_replication_dr_recovered_region 0
# HELP pd_replication_dr_state State of DR
# TYPE pd_replication_dr_state gauge
pd_replication_dr_state 3
# HELP pd_replication_dr_state_id State ID of DR
# TYPE pd_replication_dr_state_id gauge
pd_replication_dr_state_id 222
# HELP pd_replication_dr_tick_total Counter of background state check count
# TYPE pd_replication_dr_tick_total counter
pd_replication_dr_tick_total 385

sync recover

# HELP pd_replication_dr_recover_progress Progress of sync_recover process
# TYPE pd_replication_dr_recover_progress gauge
pd_replication_dr_recover_progress 0.24528302252292633
# HELP pd_replication_dr_recovered_region Number of recovered regions
# TYPE pd_replication_dr_recovered_region gauge
pd_replication_dr_recovered_region 0
# HELP pd_replication_dr_state State of DR
# TYPE pd_replication_dr_state gauge
pd_replication_dr_state 4
# HELP pd_replication_dr_state_id State ID of DR
# TYPE pd_replication_dr_state_id gauge
pd_replication_dr_state_id 223
# HELP pd_replication_dr_tick_total Counter of background state check count
# TYPE pd_replication_dr_tick_total counter
pd_replication_dr_tick_total 393

sync

# HELP pd_replication_dr_recover_progress Progress of sync_recover process
# TYPE pd_replication_dr_recover_progress gauge
pd_replication_dr_recover_progress 1
# HELP pd_replication_dr_recovered_region Number of recovered regions
# TYPE pd_replication_dr_recovered_region gauge
pd_replication_dr_recovered_region 53
# HELP pd_replication_dr_state State of DR
# TYPE pd_replication_dr_state gauge
pd_replication_dr_state 1
# HELP pd_replication_dr_state_id State ID of DR
# TYPE pd_replication_dr_state_id gauge
pd_replication_dr_state_id 224
# HELP pd_replication_dr_tick_total Counter of background state check count
# TYPE pd_replication_dr_tick_total counter
pd_replication_dr_tick_total 614

Release note

None.

fix tikv#6975 Signed-off-by: disksing <[email protected]>

ti-chi-bot · 2023-09-19T07:48:30Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

lhy1024
nolouch

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

codecov · 2023-09-19T08:07:28Z

Codecov Report

Merging #7110 (179676d) into master (3c632b1) will increase coverage by 0.02%.
Report is 1 commits behind head on master.
The diff coverage is 86.36%.

@@            Coverage Diff             @@
##           master    #7110      +/-   ##
==========================================
+ Coverage   74.40%   74.42%   +0.02%     
==========================================
  Files         440      441       +1     
  Lines       47180    47216      +36     
==========================================
+ Hits        35104    35141      +37     
- Misses       8986     8989       +3     
+ Partials     3090     3086       -4

Flag	Coverage Δ
unittests	`74.42% <86.36%> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Signed-off-by: disksing <[email protected]>

pkg/replication/metrics.go

Signed-off-by: disksing <[email protected]>

disksing · 2023-09-22T07:21:51Z

@CabinfeverB added tick rate panel.

CabinfeverB · 2023-09-22T07:34:19Z

/merge

ti-chi-bot · 2023-09-22T07:34:20Z

@CabinfeverB: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2023-09-22T07:34:22Z

This pull request has been accepted and is ready to merge.

Commit hash: 34430ea

CabinfeverB · 2023-09-22T08:07:22Z

/merge

ti-chi-bot · 2023-09-25T03:41:44Z

@disksing: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

disksing · 2023-09-25T03:49:17Z

/run-all-tests

CabinfeverB · 2023-09-25T07:18:37Z

/hold

CabinfeverB · 2023-09-25T07:20:46Z

/hold cancel

CabinfeverB · 2023-09-25T07:21:24Z

/hold

CabinfeverB · 2023-09-25T07:29:39Z

/hold cancel

disksing · 2023-09-26T02:29:15Z

/cherry-pick release-6.5

ti-chi-bot · 2023-09-26T02:29:23Z

In response to a cherrypick label: new pull request created to branch release-6.5: #7153.

ti-chi-bot · 2023-09-26T02:29:59Z

@disksing: new pull request could not be created: failed to create pull request against tikv/pd#release-6.5 from head ti-chi-bot:cherry-pick-7110-to-release-6.5: status code 422 not one of [201], body: {"message":"Validation Failed","errors":[{"resource":"PullRequest","code":"custom","message":"A pull request already exists for ti-chi-bot:cherry-pick-7110-to-release-6.5."}],"documentation_url":"https://docs.github.com/rest/pulls/pulls#create-a-pull-request"}

In response to this:

/cherry-pick release-6.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

close #6975 Signed-off-by: disksing <[email protected]> Co-authored-by: disksing <[email protected]>

close tikv#6975 Signed-off-by: disksing <[email protected]> Co-authored-by: disksing <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

TonsnakeLin · 2024-02-01T04:00:12Z

/cherry-pick release-7.1

close tikv#6975 Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot · 2024-02-01T04:00:49Z

@TonsnakeLin: new pull request created to branch release-7.1: #7785.

In response to this:

/cherry-pick release-7.1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

close tikv#6975 Co-authored-by: Ti Chi Robot <[email protected]>

close tikv#6975 Co-authored-by: Ti Chi Robot <[email protected]> Signed-off-by: TonsnakeLin <[email protected]>

close #6975 Signed-off-by: TonsnakeLin <[email protected]> Co-authored-by: disksing <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

dr-autosync: add metrics

4764721

fix tikv#6975 Signed-off-by: disksing <[email protected]>

ti-chi-bot bot added the release-note-none Denotes a PR that doesn't merit a release note. label Sep 19, 2023

Merge branch 'master' into disksing/metrics

7848895

ti-chi-bot bot requested review from lhy1024 and nolouch September 19, 2023 07:48

ti-chi-bot bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Sep 19, 2023

disksing added 2 commits September 22, 2023 10:27

Merge branch 'master' into disksing/metrics

2d8fec3

update grafana template

1737c93

Signed-off-by: disksing <[email protected]>

ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Sep 22, 2023

Merge branch 'master' into disksing/metrics

1ae6161

nolouch approved these changes Sep 22, 2023

View reviewed changes

ti-chi-bot bot added the status/LGT1 Indicates that a PR has LGTM 1. label Sep 22, 2023

CabinfeverB reviewed Sep 22, 2023

View reviewed changes

pkg/replication/metrics.go Show resolved Hide resolved

lhy1024 approved these changes Sep 22, 2023

View reviewed changes

ti-chi-bot bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Sep 22, 2023

update grafana

d9c7ac7

Signed-off-by: disksing <[email protected]>

ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Sep 22, 2023

Merge branch 'master' into disksing/metrics

34430ea

ti-chi-bot bot added the status/can-merge Indicates a PR has been approved by a committer. label Sep 22, 2023

CabinfeverB approved these changes Sep 22, 2023

View reviewed changes

ti-chi-bot bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 25, 2023

ti-chi-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 25, 2023

ti-chi-bot bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 25, 2023

Merge branch 'master' into disksing/metrics

70ac4e4

ti-chi-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 25, 2023

disksing added 2 commits September 25, 2023 15:38

Merge branch 'master' into disksing/metrics

8ede4a5

Merge branch 'master' into disksing/metrics

179676d

ti-chi-bot bot merged commit eb6953f into tikv:master Sep 25, 2023
22 checks passed

disksing added the needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. label Sep 26, 2023

ti-chi-bot mentioned this pull request Sep 26, 2023

dr-autosync: add metrics (#7110) #7153

Merged

ti-chi-bot bot pushed a commit that referenced this pull request Sep 26, 2023

dr-autosync: add metrics (#7110) (#7153)

ed629e9

close #6975 Signed-off-by: disksing <[email protected]> Co-authored-by: disksing <[email protected]>

disksing added a commit to oh-my-tidb/pd that referenced this pull request Oct 30, 2023

dr-autosync: add metrics (tikv#7110) (tikv#7153)

67e6509

close tikv#6975 Signed-off-by: disksing <[email protected]> Co-authored-by: disksing <[email protected]>

disksing added a commit that referenced this pull request Oct 30, 2023

dr-autosync: add metrics (#7110) (#7153) (#7284)

c5b59c8

Co-authored-by: Ti Chi Robot <[email protected]>

ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this pull request Feb 1, 2024

This is an automated cherry-pick of tikv#7110

3c6532b

close tikv#6975 Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Feb 1, 2024

dr-autosync: add metrics (#7110) #7785

Closed

TonsnakeLin pushed a commit to TonsnakeLin/pd that referenced this pull request Feb 28, 2024

dr-autosync: add metrics (tikv#7110)

323c945

close tikv#6975 Co-authored-by: Ti Chi Robot <[email protected]>

TonsnakeLin pushed a commit to TonsnakeLin/pd that referenced this pull request Feb 28, 2024

dr-autosync: add metrics (tikv#7110)

e336bf3

close tikv#6975 Co-authored-by: Ti Chi Robot <[email protected]> Signed-off-by: TonsnakeLin <[email protected]>

TonsnakeLin mentioned this pull request Mar 1, 2024

dr-autosync: cherry-pick add metrics (#7110) #7855

Merged

ti-chi-bot bot pushed a commit that referenced this pull request Mar 1, 2024

dr-autosync: cherry-pick add metrics (#7110) (#7855)

03296dc

close #6975 Signed-off-by: TonsnakeLin <[email protected]> Co-authored-by: disksing <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dr-autosync: add metrics #7110

dr-autosync: add metrics #7110

disksing commented Sep 19, 2023 •

edited

Loading

ti-chi-bot bot commented Sep 19, 2023 •

edited

Loading

codecov bot commented Sep 19, 2023 •

edited

Loading

disksing commented Sep 22, 2023

CabinfeverB commented Sep 22, 2023

ti-chi-bot bot commented Sep 22, 2023

ti-chi-bot bot commented Sep 22, 2023

CabinfeverB commented Sep 22, 2023

ti-chi-bot bot commented Sep 25, 2023

disksing commented Sep 25, 2023

CabinfeverB commented Sep 25, 2023

CabinfeverB commented Sep 25, 2023

CabinfeverB commented Sep 25, 2023

CabinfeverB commented Sep 25, 2023

disksing commented Sep 26, 2023

ti-chi-bot commented Sep 26, 2023

ti-chi-bot commented Sep 26, 2023

TonsnakeLin commented Feb 1, 2024

ti-chi-bot commented Feb 1, 2024

dr-autosync: add metrics #7110

dr-autosync: add metrics #7110

Conversation

disksing commented Sep 19, 2023 • edited Loading

What problem does this PR solve?

What is changed and how does it work?

Check List

Release note

ti-chi-bot bot commented Sep 19, 2023 • edited Loading

codecov bot commented Sep 19, 2023 • edited Loading

Codecov Report

disksing commented Sep 22, 2023

CabinfeverB commented Sep 22, 2023

ti-chi-bot bot commented Sep 22, 2023

ti-chi-bot bot commented Sep 22, 2023

CabinfeverB commented Sep 22, 2023

ti-chi-bot bot commented Sep 25, 2023

disksing commented Sep 25, 2023

CabinfeverB commented Sep 25, 2023

CabinfeverB commented Sep 25, 2023

CabinfeverB commented Sep 25, 2023

CabinfeverB commented Sep 25, 2023

disksing commented Sep 26, 2023

ti-chi-bot commented Sep 26, 2023

ti-chi-bot commented Sep 26, 2023

TonsnakeLin commented Feb 1, 2024

ti-chi-bot commented Feb 1, 2024

disksing commented Sep 19, 2023 •

edited

Loading

ti-chi-bot bot commented Sep 19, 2023 •

edited

Loading

codecov bot commented Sep 19, 2023 •

edited

Loading