Add wlm resiliency orchestrator (query group service) #15925

kaushalmahi12 · 2024-09-13T01:13:31Z

Description

This change adds the last piece in WLM resiliency feature (Query Sandboxing). Post this PR we will be raising the ITs for the feature.

Related Issues

Main RFC: #12342

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Kiran Prakash <[email protected]>

Signed-off-by: Kaushal Kumar <[email protected]>

github-actions · 2024-10-01T01:31:30Z

❌ Gradle check result for 56c3393: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Kaushal Kumar <[email protected]>

github-actions · 2024-10-01T22:05:22Z

❌ Gradle check result for 3c6adae: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

jainankitk

@kaushalmahi12 - Looks much better now. Few minor comments, should be ready to merge soon!

server/src/main/java/org/opensearch/wlm/QueryGroupTask.java

server/src/main/java/org/opensearch/wlm/WorkloadManagementSettings.java

server/src/main/java/org/opensearch/wlm/cancellation/QueryGroupTaskCancellationService.java

server/src/main/java/org/opensearch/wlm/stats/QueryGroupStats.java

jainankitk · 2024-10-04T06:57:09Z

Also, can you merge from main branch to resolve the conflict in server/src/main/java/org/opensearch/common/settings/ClusterSettings.java

Signed-off-by: Kaushal Kumar <[email protected]>

server/src/main/java/org/opensearch/wlm/QueryGroupService.java

github-actions · 2024-10-07T18:24:24Z

✅ Gradle check result for 862af7b: SUCCESS

Signed-off-by: Kaushal Kumar <[email protected]>

github-actions · 2024-10-08T00:02:48Z

❕ Gradle check result for 2d7316b: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

opensearch-trigger-bot · 2024-10-08T00:22:06Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-15925-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 e24b4c98405d95b781a43fe73ca370c1a1ca452a
# Push it to GitHub
git push --set-upstream origin backport/backport-15925-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-15925-to-2.x.

…ject#15925) * cancellation related Signed-off-by: Kiran Prakash <[email protected]> * Update CHANGELOG.md Signed-off-by: Kiran Prakash <[email protected]> * add better cancellation reason Signed-off-by: Kiran Prakash <[email protected]> * Update DefaultTaskCancellationTests.java Signed-off-by: Kiran Prakash <[email protected]> * refactor Signed-off-by: Kiran Prakash <[email protected]> * refactor Signed-off-by: Kiran Prakash <[email protected]> * Update DefaultTaskCancellation.java Signed-off-by: Kiran Prakash <[email protected]> * Update DefaultTaskCancellation.java Signed-off-by: Kiran Prakash <[email protected]> * Update DefaultTaskCancellation.java Signed-off-by: Kiran Prakash <[email protected]> * Update DefaultTaskSelectionStrategy.java Signed-off-by: Kiran Prakash <[email protected]> * refactor Signed-off-by: Kiran Prakash <[email protected]> * refactor node level threshold Signed-off-by: Kiran Prakash <[email protected]> * use query group task Signed-off-by: Kaushal Kumar <[email protected]> * code clean up and refactorings Signed-off-by: Kaushal Kumar <[email protected]> * add unit tests and fix existing ones Signed-off-by: Kaushal Kumar <[email protected]> * uncomment the test case Signed-off-by: Kaushal Kumar <[email protected]> * update CHANGELOG Signed-off-by: Kaushal Kumar <[email protected]> * fix imports Signed-off-by: Kaushal Kumar <[email protected]> * add queryGroupService Signed-off-by: Kaushal Kumar <[email protected]> * refactor and add UTs for new constructs Signed-off-by: Kaushal Kumar <[email protected]> * fix javadocs Signed-off-by: Kaushal Kumar <[email protected]> * remove code clutter Signed-off-by: Kaushal Kumar <[email protected]> * change annotation version and task selection strategy Signed-off-by: Kaushal Kumar <[email protected]> * rename a util class Signed-off-by: Kaushal Kumar <[email protected]> * remove wrappers from resource type Signed-off-by: Kaushal Kumar <[email protected]> * apply spotless Signed-off-by: Kaushal Kumar <[email protected]> * address comments Signed-off-by: Kaushal Kumar <[email protected]> * add rename changes Signed-off-by: Kaushal Kumar <[email protected]> * address comments Signed-off-by: Kaushal Kumar <[email protected]> * initial changes Signed-off-by: Kaushal Kumar <[email protected]> * refactor changes and logical bug fix Signed-off-by: Kaushal Kumar <[email protected]> * add chanegs Signed-off-by: Kaushal Kumar <[email protected]> * address comments Signed-off-by: Kaushal Kumar <[email protected]> * temp changes Signed-off-by: Kaushal Kumar <[email protected]> * add UTs Signed-off-by: Kaushal Kumar <[email protected]> * add changelog Signed-off-by: Kaushal Kumar <[email protected]> * add task completion listener hook Signed-off-by: Kaushal Kumar <[email protected]> * add remaining pieces to make the feature functional Signed-off-by: Kaushal Kumar <[email protected]> * extend stats and fix bugs Signed-off-by: Kaushal Kumar <[email protected]> * fix bugs and add logic to make SBP work with wlm Signed-off-by: Kaushal Kumar <[email protected]> * address comments Signed-off-by: Kaushal Kumar <[email protected]> * fix bugs and SBP ITs Signed-off-by: Kaushal Kumar <[email protected]> * add missed applyCluster state change Signed-off-by: Kaushal Kumar <[email protected]> * address comments Signed-off-by: Kaushal Kumar <[email protected]> * decouple queryGroupService and cancellationService Signed-off-by: Kaushal Kumar <[email protected]> * replace StateApplier with StateListener interface Signed-off-by: Kaushal Kumar <[email protected]> * fix precommit errors Signed-off-by: Kaushal Kumar <[email protected]> --------- Signed-off-by: Kiran Prakash <[email protected]> Signed-off-by: Kaushal Kumar <[email protected]> Co-authored-by: Kiran Prakash <[email protected]>

* cancellation related * Update CHANGELOG.md * add better cancellation reason * Update DefaultTaskCancellationTests.java * refactor * refactor * Update DefaultTaskCancellation.java * Update DefaultTaskCancellation.java * Update DefaultTaskCancellation.java * Update DefaultTaskSelectionStrategy.java * refactor * refactor node level threshold * use query group task * code clean up and refactorings * add unit tests and fix existing ones * uncomment the test case * update CHANGELOG * fix imports * add queryGroupService * refactor and add UTs for new constructs * fix javadocs * remove code clutter * change annotation version and task selection strategy * rename a util class * remove wrappers from resource type * apply spotless * address comments * add rename changes * address comments * initial changes * refactor changes and logical bug fix * add chanegs * address comments * temp changes * add UTs * add changelog * add task completion listener hook * add remaining pieces to make the feature functional * extend stats and fix bugs * fix bugs and add logic to make SBP work with wlm * address comments * fix bugs and SBP ITs * add missed applyCluster state change * address comments * decouple queryGroupService and cancellationService * replace StateApplier with StateListener interface * fix precommit errors --------- Signed-off-by: Kiran Prakash <[email protected]> Signed-off-by: Kaushal Kumar <[email protected]> Co-authored-by: Kiran Prakash <[email protected]>

…ject#15925) * cancellation related Signed-off-by: Kiran Prakash <[email protected]> * Update CHANGELOG.md Signed-off-by: Kiran Prakash <[email protected]> * add better cancellation reason Signed-off-by: Kiran Prakash <[email protected]> * Update DefaultTaskCancellationTests.java Signed-off-by: Kiran Prakash <[email protected]> * refactor Signed-off-by: Kiran Prakash <[email protected]> * refactor Signed-off-by: Kiran Prakash <[email protected]> * Update DefaultTaskCancellation.java Signed-off-by: Kiran Prakash <[email protected]> * Update DefaultTaskCancellation.java Signed-off-by: Kiran Prakash <[email protected]> * Update DefaultTaskCancellation.java Signed-off-by: Kiran Prakash <[email protected]> * Update DefaultTaskSelectionStrategy.java Signed-off-by: Kiran Prakash <[email protected]> * refactor Signed-off-by: Kiran Prakash <[email protected]> * refactor node level threshold Signed-off-by: Kiran Prakash <[email protected]> * use query group task Signed-off-by: Kaushal Kumar <[email protected]> * code clean up and refactorings Signed-off-by: Kaushal Kumar <[email protected]> * add unit tests and fix existing ones Signed-off-by: Kaushal Kumar <[email protected]> * uncomment the test case Signed-off-by: Kaushal Kumar <[email protected]> * update CHANGELOG Signed-off-by: Kaushal Kumar <[email protected]> * fix imports Signed-off-by: Kaushal Kumar <[email protected]> * add queryGroupService Signed-off-by: Kaushal Kumar <[email protected]> * refactor and add UTs for new constructs Signed-off-by: Kaushal Kumar <[email protected]> * fix javadocs Signed-off-by: Kaushal Kumar <[email protected]> * remove code clutter Signed-off-by: Kaushal Kumar <[email protected]> * change annotation version and task selection strategy Signed-off-by: Kaushal Kumar <[email protected]> * rename a util class Signed-off-by: Kaushal Kumar <[email protected]> * remove wrappers from resource type Signed-off-by: Kaushal Kumar <[email protected]> * apply spotless Signed-off-by: Kaushal Kumar <[email protected]> * address comments Signed-off-by: Kaushal Kumar <[email protected]> * add rename changes Signed-off-by: Kaushal Kumar <[email protected]> * address comments Signed-off-by: Kaushal Kumar <[email protected]> * initial changes Signed-off-by: Kaushal Kumar <[email protected]> * refactor changes and logical bug fix Signed-off-by: Kaushal Kumar <[email protected]> * add chanegs Signed-off-by: Kaushal Kumar <[email protected]> * address comments Signed-off-by: Kaushal Kumar <[email protected]> * temp changes Signed-off-by: Kaushal Kumar <[email protected]> * add UTs Signed-off-by: Kaushal Kumar <[email protected]> * add changelog Signed-off-by: Kaushal Kumar <[email protected]> * add task completion listener hook Signed-off-by: Kaushal Kumar <[email protected]> * add remaining pieces to make the feature functional Signed-off-by: Kaushal Kumar <[email protected]> * extend stats and fix bugs Signed-off-by: Kaushal Kumar <[email protected]> * fix bugs and add logic to make SBP work with wlm Signed-off-by: Kaushal Kumar <[email protected]> * address comments Signed-off-by: Kaushal Kumar <[email protected]> * fix bugs and SBP ITs Signed-off-by: Kaushal Kumar <[email protected]> * add missed applyCluster state change Signed-off-by: Kaushal Kumar <[email protected]> * address comments Signed-off-by: Kaushal Kumar <[email protected]> * decouple queryGroupService and cancellationService Signed-off-by: Kaushal Kumar <[email protected]> * replace StateApplier with StateListener interface Signed-off-by: Kaushal Kumar <[email protected]> * fix precommit errors Signed-off-by: Kaushal Kumar <[email protected]> --------- Signed-off-by: Kiran Prakash <[email protected]> Signed-off-by: Kaushal Kumar <[email protected]> Co-authored-by: Kiran Prakash <[email protected]>

kiranprakash154 and others added 30 commits August 28, 2024 06:14

cancellation related

a720136

Signed-off-by: Kiran Prakash <[email protected]>

Update CHANGELOG.md

83e20c0

Signed-off-by: Kiran Prakash <[email protected]>

add better cancellation reason

9983c73

Signed-off-by: Kiran Prakash <[email protected]>

Update DefaultTaskCancellationTests.java

245ee5d

Signed-off-by: Kiran Prakash <[email protected]>

refactor

0771fd2

Signed-off-by: Kiran Prakash <[email protected]>

refactor

4b1ef81

Signed-off-by: Kiran Prakash <[email protected]>

Update DefaultTaskCancellation.java

3ea44d7

Signed-off-by: Kiran Prakash <[email protected]>

Update DefaultTaskCancellation.java

0103089

Signed-off-by: Kiran Prakash <[email protected]>

Update DefaultTaskCancellation.java

092d715

Signed-off-by: Kiran Prakash <[email protected]>

Update DefaultTaskSelectionStrategy.java

4a2c51e

Signed-off-by: Kiran Prakash <[email protected]>

refactor

cbb51bd

Signed-off-by: Kiran Prakash <[email protected]>

refactor node level threshold

4e846e2

Signed-off-by: Kiran Prakash <[email protected]>

Merge branch 'main' into kp/wlm-cancellation-1

241b036

use query group task

7511d99

Signed-off-by: Kaushal Kumar <[email protected]>

code clean up and refactorings

498743a

Signed-off-by: Kaushal Kumar <[email protected]>

add unit tests and fix existing ones

e26e525

Signed-off-by: Kaushal Kumar <[email protected]>

uncomment the test case

0ff2b09

Signed-off-by: Kaushal Kumar <[email protected]>

update CHANGELOG

ddb8dce

Signed-off-by: Kaushal Kumar <[email protected]>

Merge branch 'main' into feature/wlm-cancellation

3528054

Signed-off-by: Kaushal Kumar <[email protected]>

fix imports

e8366a5

Signed-off-by: Kaushal Kumar <[email protected]>

add queryGroupService

d2d02e3

Signed-off-by: Kaushal Kumar <[email protected]>

refactor and add UTs for new constructs

448ea41

Signed-off-by: Kaushal Kumar <[email protected]>

fix javadocs

3fc21be

Signed-off-by: Kaushal Kumar <[email protected]>

remove code clutter

fe02a6a

Signed-off-by: Kaushal Kumar <[email protected]>

change annotation version and task selection strategy

8aede33

Signed-off-by: Kaushal Kumar <[email protected]>

rename a util class

623f6f8

Signed-off-by: Kaushal Kumar <[email protected]>

remove wrappers from resource type

9e2e3ea

Signed-off-by: Kaushal Kumar <[email protected]>

apply spotless

34184ef

Signed-off-by: Kaushal Kumar <[email protected]>

address comments

91893e7

Signed-off-by: Kaushal Kumar <[email protected]>

add rename changes

66e43b2

Signed-off-by: Kaushal Kumar <[email protected]>

opensearch-ci-bot mentioned this pull request Oct 1, 2024

[AUTOCUT] Gradle Check Flaky Test Report for SearchRestCancellationIT #14311

Open

kaushalmahi12 added 2 commits October 1, 2024 14:23

fix bugs and SBP ITs

baa39c5

Signed-off-by: Kaushal Kumar <[email protected]>

add missed applyCluster state change

3c6adae

Signed-off-by: Kaushal Kumar <[email protected]>

jainankitk reviewed Oct 4, 2024

View reviewed changes

kaushalmahi12 added 2 commits October 7, 2024 10:12

Merge branch 'main' into feature/wlm-service

040ed4c

Signed-off-by: Kaushal Kumar <[email protected]>

address comments

862af7b

Signed-off-by: Kaushal Kumar <[email protected]>

jainankitk reviewed Oct 7, 2024

View reviewed changes

kaushalmahi12 added 3 commits October 7, 2024 15:24

decouple queryGroupService and cancellationService

e2b203c

Signed-off-by: Kaushal Kumar <[email protected]>

replace StateApplier with StateListener interface

623078a

Signed-off-by: Kaushal Kumar <[email protected]>

fix precommit errors

2d7316b

Signed-off-by: Kaushal Kumar <[email protected]>

jainankitk approved these changes Oct 8, 2024

View reviewed changes

jainankitk merged commit e24b4c9 into opensearch-project:main Oct 8, 2024
34 of 35 checks passed

opensearch-trigger-bot bot added the backport-failed label Oct 8, 2024

opensearch-ci-bot mentioned this pull request Oct 8, 2024

[AUTOCUT] Gradle Check Flaky Test Report for RemoteStoreIT #16145

Open

kaushalmahi12 mentioned this pull request Oct 8, 2024

[Backport 2.x] Add wlm resiliency orchestrator (query group service) (#15925) #16225

Merged

This was referenced Oct 8, 2024

[AUTOCUT] Gradle Check Flaky Test Report for ApproximatePointRangeQueryTests #15807

Open

[AUTOCUT] Gradle Check Flaky Test Report for ReindexFailureTests #16286

Open

opensearch-ci-bot mentioned this pull request Oct 23, 2024

[AUTOCUT] Gradle Check Flaky Test Report for UpdateByQueryBasicTests #16439

Open

BrewTestBot mentioned this pull request Nov 6, 2024

opensearch 2.18.0 Homebrew/homebrew-core#196785

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add wlm resiliency orchestrator (query group service) #15925

Add wlm resiliency orchestrator (query group service) #15925

kaushalmahi12 commented Sep 13, 2024 •

edited

Loading

github-actions bot commented Oct 1, 2024

github-actions bot commented Oct 1, 2024

jainankitk left a comment

jainankitk commented Oct 4, 2024

github-actions bot commented Oct 7, 2024

github-actions bot commented Oct 8, 2024

opensearch-trigger-bot bot commented Oct 8, 2024

Add wlm resiliency orchestrator (query group service) #15925

Add wlm resiliency orchestrator (query group service) #15925

Conversation

kaushalmahi12 commented Sep 13, 2024 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Oct 1, 2024

github-actions bot commented Oct 1, 2024

jainankitk left a comment

Choose a reason for hiding this comment

jainankitk commented Oct 4, 2024

github-actions bot commented Oct 7, 2024

github-actions bot commented Oct 8, 2024

opensearch-trigger-bot bot commented Oct 8, 2024

kaushalmahi12 commented Sep 13, 2024 •

edited

Loading