schedule: avoid data race and panic about scheduler controller

[lhy1024]

What problem does this PR solve?

Issue Number: Close #7106

What is changed and how does it work?

add a flexible wait group, because WaitGroup must ensure previous Wait has returned before added again.

Check List

Tests

• Unit test

Release note

None.

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[lhy1024]

/check-issue-triage-complete

[codecov]

Codecov Report

Merging #7107 (55adf55) into master (eb6953f) will increase coverage by 0.12%.
Report is 1 commits behind head on master.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #7107      +/-   ##
==========================================
+ Coverage   74.26%   74.38%   +0.12%     
==========================================
  Files         441      442       +1     
  Lines       47216    47234      +18     
==========================================
+ Hits        35064    35135      +71     
+ Misses       9063     9003      -60     
- Partials     3089     3096       +7     

Flag	Coverage Δ
unittests	74.38% <100.00%> (+0.12%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

[HuSharp]

seems like DATA RACE in ci
https://github.com/tikv/pd/actions/runs/6252103050/job/16974605966?pr=7107

==================
WARNING: DATA RACE
Write at 0x00c0000b0098 by goroutine 37:
  github.com/tikv/pd/pkg/utils/syncutil.(*FlexibleWaitGroup).Add()
      /home/runner/work/pd/pd/pkg/utils/syncutil/flexible_wait_group.go:43 +0x107
  github.com/tikv/pd/pkg/utils/syncutil.TestNegativeDelta.func2()
      /home/runner/work/pd/pd/pkg/utils/syncutil/flexible_wait_group_test.go:82 +0x[35](https://github.com/tikv/pd/actions/runs/6252103050/job/16974605966?pr=7107#step:5:36)

Previous read at 0x00c0000b0098 by goroutine 35:
  github.com/tikv/pd/pkg/utils/syncutil.TestNegativeDelta()
      /home/runner/work/pd/pd/pkg/utils/syncutil/flexible_wait_group_test.go:86 +0x224
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.1/x64/src/testing/testing.go:1595 +0x238
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.1/x64/src/testing/testing.go:1648 +0x44

Goroutine [37](https://github.com/tikv/pd/actions/runs/6252103050/job/16974605966?pr=7107#step:5:38) (running) created at:
  github.com/tikv/pd/pkg/utils/syncutil.TestNegativeDelta()
      /home/runner/work/pd/pd/pkg/utils/syncutil/flexible_wait_group_test.go:81 +0x205
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.1/x64/src/testing/testing.go:1595 +0x2[38](https://github.com/tikv/pd/actions/runs/6252103050/job/16974605966?pr=7107#step:5:39)
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.1/x64/src/testing/testing.go:1648 +0x[44](https://github.com/tikv/pd/actions/runs/6252103050/job/16974605966?pr=7107#step:5:45)```

[nolouch]

will it always hold the lock?

[lhy1024]

No, fwg.cond.Wait() will work