Initial commit to support a search only replica for RW separation.

[mch2]

Description

Initial commit for the search only replica for reader writer split.
This PR contains the following:

1. Introduce searchOnly flag on ShardRouting.
2. Add feature flag to enable/disable the feature.
3. support both create and update APIs to toggle search replica count.
4. Changes to exclude search replicas from primary eligibility.
5. Changes to prevent any replicationOperation from routing to a search replica.
6. Small change to return [s] instead of [r] for search replicas in /_cat/shards

Related Issues

Resolves #15368
related #15306

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

✅ Gradle check result for e182c9e: SUCCESS

[mch2]

❌ Gradle check result for c5d29a9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

#10820

[github-actions]

✅ Gradle check result for 2207a10: SUCCESS

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-15410-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 1e9fdb452101476a24737cc1b3aa1fad15df8fca
# Push it to GitHub
git push --set-upstream origin backport/backport-15410-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-15410-to-2.x.

[Bukhtawar]

Sorry couldn't understand the context for this change. This is meant to move the replica and check if the outgoing recovery from it's peer primary is getting throttled. Wouldn't search only replica recover from the primary copy or the plan is to make recoveries totally independent? cc: @mch2

[mch2]

Err thanks for calling this out I'm not handling this correclty.

This was to resolve an NPE a few lines later when we only have search replicas and the primary has a node drop. In this case the NPE occurs right after this where we fetch the active primary as null and then invoke .currentNodeId on it. We should be setting outgoingRecoveries to 0 in this case instead.

To answer your question the plan is to have search replica recover directly from the remote store when its enabled but we need to still honor node-node with active primary.