tests: add frequency check time to make resign test more stable

[HuSharp]

What problem does this PR solve?

tests: add frequency check time to make resign test more stable

--- FAIL: TestRegionStatistics (17.47s)
    cluster_test.go:262: 
        	Error Trace:	/home/runner/work/pd/pd/tests/server/cluster/cluster_test.go:262
        	Error:      	Should not be: "pd3"
        	Test:       	TestRegionStatistics

tikv/pd/actions/runs/9967757112/job/27541904077

Conclude: Because frequently change(There are many resign leader in this test and slow disk can make frequently happened in high probability :(

cluster_test.go:262 checks pd leader not be pd3, which means cluster.ResignLeader() regards pd3 is leader, but tc.WaitLeader() still get pd3

	leaderName = leaderServer.GetServer().Name()
	leaderServer.ResignLeader()
	re.NotEqual(tc.WaitLeader(), leaderName)

Let's check the log:

// now leader is pd3
[2024/07/17 03:38:56.625 +00:00] [INFO] [server.go:1737] ["campaign PD leader ok"] [campaign-leader-name=pd3]
[2024/07/17 03:38:57.143 +00:00] [INFO] [member.go:356] ["try to resign etcd leader to next pd-server"] [from=pd3] [to=]
// due to frequently transfer leader to pd3 as well
[2024/07/17 03:38:58.149 +00:00] [INFO] [member.go:356] ["try to resign etcd leader to next pd-server"] [from=pd2] [to=]
[2024/07/17 03:38:58.190 +00:00] [INFO] [server.go:1737] ["campaign PD leader ok"] [campaign-leader-name=pd3]
[2024/07/17 03:38:58.650 +00:00] [ERROR] [server.go:1717] ["campaign PD leader meets error due to etcd error"] [campaign-leader-name=pd2] [error="[PD:server:ErrLeaderFrequentlyChange]leader pd2 frequently changed, leader-key is [/pd/7392444153452143012/leader]"]

Issue Number: Close #8319, #8351

What is changed and how does it work?

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)

Release note

None.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 77.32%. Comparing base (c514937) to head (9f2ee7f).
Report is 1 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #8411   +/-   ##
=======================================
  Coverage   77.31%   77.32%           
=======================================
  Files         471      471           
  Lines       61878    61882    +4     
=======================================
+ Hits        47840    47849    +9     
+ Misses      10466    10457    -9     
- Partials     3572     3576    +4     

Flag	Coverage Δ
unittests	77.32% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

[ti-chi-bot]

@okJiang: adding LGTM is restricted to approvers and reviewers in OWNERS files.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[JmPotato]

What about using failpoint here?

[HuSharp]

changed

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JmPotato, lhy1024, okJiang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [JmPotato,lhy1024]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-07-17 08:28:39.428472197 +0000 UTC m=+428941.419413668: ☑️ agreed by lhy1024.
• 2024-07-19 10:11:41.278971345 +0000 UTC m=+607923.269912815: ☑️ agreed by JmPotato.