Fix Flaky Test SpecificClusterManagerNodesIT.testElectOnlyBetweenClusterManagerNodes

[kkewwei]

Description

The case is as follows:

1. When the node_t1 is excluded from the vote config, and the cluster starts a new leader election, but the the node node_t2 hasn't been elected as the new leader.

2. At the moment, we send request to get the ClusterManager, we first get ClusterManager name, and leads to the NullPointerException.

internalCluster().nonClusterManagerClient()-> ......->getClusterManagerName()

OpenSearch/test/framework/src/main/java/org/opensearch/test/InternalTestCluster.java

Line 2171 in f0ea056

return client.admin().cluster().prepareState().get().getState().nodes().getClusterManagerNode().getName();

Related Issues

Resolves #15944 #16015

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❕ Gradle check result for 1b3920b: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[msfroh]

Checking the history of this test, has this been flaky for more than a year?

The only code related to cluster manager election that I've been able to find changed more recently than ~5 years ago (besides renaming) is the introduction of DecommisionService in 2023.

[kkewwei]

Checking the history of this test, has this been flaky for more than a year?

The only code related to cluster manager election that I've been able to find changed more recently than ~5 years ago (besides renaming) is the introduction of DecommisionService in 2023.

@msfroh I'm not clearly why it wasn’t reported before, maybe the frequency of occurrence is very low and we ignored it? This issue #15944 collects some of the recent case.

Logically speaking, when the cluster has no cluster manager, It's right to return null from getClusterManagerNode, which will hit the case.

[github-actions]

✅ Gradle check result for 4d812e9: SUCCESS

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.98%. Comparing base (036f6bc) to head (4d812e9).
Report is 239 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #16021      +/-   ##
============================================
+ Coverage     71.92%   71.98%   +0.06%     
  Complexity    64400    64400              
============================================
  Files          5281     5281              
  Lines        300995   300995              
  Branches      43479    43479              
============================================
+ Hits         216491   216672     +181     
+ Misses        66793    66569     -224     
- Partials      17711    17754      +43     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[kkewwei]

@msfroh @gaobinlong please help confirm it when you are free.

[opensearch-trigger-bot]

This PR is stalled because it has been open for 30 days with no activity.

[jed326]

So if we swallow the NPE with the try/catch above, won't this test still fail on the assertion here since clusterManagerName.get() would be null and nextClusterManagerEligableNodeName shouldn't be null?

[kkewwei]

@jed326 assertBusy allows the assert failed in 10s, so it seems ok.