Change Remote state read thread pool to Fixed type

[soosinha]

Description

Remote state read thread pool is of Scaling type. So if for any reason multiple full cluster state downloads are attempted, it will keep filling up the queue and can cause other issues. So changing Remote state read thread pool to Fixed type with limit as queue size as 120,000. The queue size is chosen asssuming that there can be max of 50k indices. So total number of files will be around 100k (50k for index metadata and 50k for routing table). Adding some buffer to 100k and making it 120k.

Also, adding some more logs to understand when full state is downloaded.

Related Issues

NA

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

✅ Gradle check result for 1c55c53: SUCCESS

[codecov]

Codecov Report

Attention: Patch coverage is 75.00000% with 4 lines in your changes missing coverage. Please review.

Project coverage is 72.21%. Comparing base (b359dd8) to head (ab77523).
Report is 9 commits behind head on main.

Files with missing lines	Patch %	Lines
...arch/gateway/remote/RemoteClusterStateService.java	62.50%	3 Missing ⚠️
...stermanager/TransportClusterManagerNodeAction.java	80.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #16850      +/-   ##
============================================
+ Coverage     72.08%   72.21%   +0.12%     
- Complexity    65279    65323      +44     
============================================
  Files          5318     5318              
  Lines        304056   304068      +12     
  Branches      43992    43992              
============================================
+ Hits         219188   219571     +383     
+ Misses        66932    66503     -429     
- Partials      17936    17994      +58     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

❕ Gradle check result for ab77523: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-16850-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 8aa3185e18bc725c1aa74ee55dcdf3c0cdc512e3
# Push it to GitHub
git push --set-upstream origin backport/backport-16850-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-16850-to-2.x.