Pagination in Hybrid query #1048

vibrantvarun · 2024-12-31T01:52:26Z

Description

This PR contains changes for enabling support for pagination in hybrid query.
The highlight of this PR are

Introduction of a new parameter "pagination_depth" to set a reference of hybrid query search results on which pagination can be applied.
Handling of single shard scenario where fetch phase can run before the normalization process.
Handling of from parameter conditions in Normalization processor.
Disabling scroll operation in hybrid query.

Related Issues

Check List

New functionality includes testing.
New functionality has been documented.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Varun Jain <[email protected]>

vibrantvarun · 2025-01-02T21:37:09Z

Integ test and BWC tests are failing due to opensearch-project/ml-commons#3321

src/main/java/org/opensearch/neuralsearch/processor/NormalizationProcessorWorkflow.java

src/main/java/org/opensearch/neuralsearch/query/HybridQuery.java

src/main/java/org/opensearch/neuralsearch/search/query/HybridCollectorManager.java

src/main/java/org/opensearch/neuralsearch/query/HybridQuery.java

Signed-off-by: Varun Jain <[email protected]>

src/main/java/org/opensearch/neuralsearch/query/HybridQueryContext.java

src/main/java/org/opensearch/neuralsearch/search/query/HybridCollectorManager.java

src/main/java/org/opensearch/neuralsearch/query/HybridQueryContext.java

… review Signed-off-by: Varun Jain <[email protected]>

Signed-off-by: Varun Jain <[email protected]>

martin-gaievski

minor changes requested, overall code looks good to me

src/main/java/org/opensearch/neuralsearch/query/HybridQueryBuilder.java

src/main/java/org/opensearch/neuralsearch/search/query/HybridCollectorManager.java

Signed-off-by: Varun Jain <[email protected]>

vibrantvarun · 2025-01-08T21:49:51Z

src/main/java/org/opensearch/neuralsearch/search/query/HybridQueryPhaseSearcher.java

@@ -78,16 +73,9 @@ private QueryPhaseSearcher getQueryPhaseSearcher(final SearchContext searchConte
            : defaultQueryPhaseSearcherWithEmptyCollectorContext;
    }

-    private static boolean isWrappedHybridQuery(final Query query) {


Duplicate method in HybridQueryUtil so removed it

Signed-off-by: Varun Jain <[email protected]>

src/main/java/org/opensearch/neuralsearch/query/HybridQueryBuilder.java

Signed-off-by: Varun Jain <[email protected]>

martin-gaievski

great work Varun, approving

src/main/java/org/opensearch/neuralsearch/query/HybridQuery.java

src/main/java/org/opensearch/neuralsearch/query/HybridQueryBuilder.java

CHANGELOG.md

Signed-off-by: Varun Jain <[email protected]>

src/main/java/org/opensearch/neuralsearch/query/HybridQueryBuilder.java

Signed-off-by: Varun Jain <[email protected]>

VijayanB

LGTM

Signed-off-by: Varun Jain <[email protected]>

opensearch-trigger-bot · 2025-01-11T16:03:25Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-1048-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 b084838490ad90f8715c0b465f08eca62623c74f
# Push it to GitHub
git push --set-upstream origin backport/backport-1048-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-1048-to-2.x.

* Pagination in Hybrid query Signed-off-by: Varun Jain <[email protected]> * Remove unwanted code Signed-off-by: Varun Jain <[email protected]> * Adding hybrid query context dto Signed-off-by: Varun Jain <[email protected]> * Adding javadoc in hybridquerycontext and addressing few comments from review Signed-off-by: Varun Jain <[email protected]> * rename hybrid query extraction method Signed-off-by: Varun Jain <[email protected]> * Refactoring to optimize extractHybridQuery method calls Signed-off-by: Varun Jain <[email protected]> * Changes in tests to adapt with builder pattern in querybuilder Signed-off-by: Varun Jain <[email protected]> * Add mapper service mock in tests Signed-off-by: Varun Jain <[email protected]> * Fix error message of index.max_result_window setting Signed-off-by: Varun Jain <[email protected]> * Fix error message of index.max_result_window setting Signed-off-by: Varun Jain <[email protected]> * Fixing validation condition for lower bound Signed-off-by: Varun Jain <[email protected]> * fix tests Signed-off-by: Varun Jain <[email protected]> * Removing version check from doEquals and doHashCode method Signed-off-by: Varun Jain <[email protected]> --------- Signed-off-by: Varun Jain <[email protected]>

* Pagination in Hybrid query (#1048) * Pagination in Hybrid query Signed-off-by: Varun Jain <[email protected]> * Remove unwanted code Signed-off-by: Varun Jain <[email protected]> * Adding hybrid query context dto Signed-off-by: Varun Jain <[email protected]> * Adding javadoc in hybridquerycontext and addressing few comments from review Signed-off-by: Varun Jain <[email protected]> * rename hybrid query extraction method Signed-off-by: Varun Jain <[email protected]> * Refactoring to optimize extractHybridQuery method calls Signed-off-by: Varun Jain <[email protected]> * Changes in tests to adapt with builder pattern in querybuilder Signed-off-by: Varun Jain <[email protected]> * Add mapper service mock in tests Signed-off-by: Varun Jain <[email protected]> * Fix error message of index.max_result_window setting Signed-off-by: Varun Jain <[email protected]> * Fix error message of index.max_result_window setting Signed-off-by: Varun Jain <[email protected]> * Fixing validation condition for lower bound Signed-off-by: Varun Jain <[email protected]> * fix tests Signed-off-by: Varun Jain <[email protected]> * Removing version check from doEquals and doHashCode method Signed-off-by: Varun Jain <[email protected]> --------- Signed-off-by: Varun Jain <[email protected]> * Update pagination_depth datatype from int to Integer (#1094) * Update pagination_depth datatype from int to Integer Signed-off-by: Varun Jain <[email protected]> --------- Signed-off-by: Varun Jain <[email protected]>

vibrantvarun added 2 commits December 30, 2024 17:48

Pagination in Hybrid query

8008a0b

Signed-off-by: Varun Jain <[email protected]>

Remove unwanted code

db56f5a

Signed-off-by: Varun Jain <[email protected]>

vibrantvarun marked this pull request as ready for review January 2, 2025 17:37

vibrantvarun requested review from heemin32, navneet1v, VijayanB, vamshin, jmazanec15, naveentatikonda, junqiu-lei, martin-gaievski, sean-zheng-amazon, model-collapse, zane-neo, zhichao-aws, yuye-aws and minalsha as code owners January 2, 2025 17:37

martin-gaievski reviewed Jan 3, 2025

View reviewed changes

vibrantvarun added 2 commits January 3, 2025 14:30

Adding hybrid query context dto

8fa8fc6

Signed-off-by: Varun Jain <[email protected]>

Merge branch 'main' into pagination

a07b714

vibrantvarun requested a review from martin-gaievski January 3, 2025 22:34

martin-gaievski reviewed Jan 4, 2025

View reviewed changes

Adding javadoc in hybridquerycontext and addressing few comments from…

1474905

… review Signed-off-by: Varun Jain <[email protected]>

vibrantvarun requested a review from martin-gaievski January 6, 2025 18:04

rename hybrid query extraction method

d39a1b8

Signed-off-by: Varun Jain <[email protected]>

vibrantvarun mentioned this pull request Jan 6, 2025

[DOC] Adding pagination in Hybrid query opensearch-project/documentation-website#8952

Open

4 tasks

martin-gaievski reviewed Jan 8, 2025

View reviewed changes

src/main/java/org/opensearch/neuralsearch/query/HybridQueryBuilder.java Outdated Show resolved Hide resolved

src/main/java/org/opensearch/neuralsearch/search/query/HybridCollectorManager.java Show resolved Hide resolved

Refactoring to optimize extractHybridQuery method calls

62498ad

Signed-off-by: Varun Jain <[email protected]>

vibrantvarun commented Jan 8, 2025

View reviewed changes

vibrantvarun added 2 commits January 8, 2025 15:55

Merge branch 'main' into pagination

195c966

Fix error message of index.max_result_window setting

1c0bdeb

Signed-off-by: Varun Jain <[email protected]>

vibrantvarun requested a review from martin-gaievski January 9, 2025 00:06

martin-gaievski reviewed Jan 9, 2025

View reviewed changes

src/main/java/org/opensearch/neuralsearch/query/HybridQueryBuilder.java Outdated Show resolved Hide resolved

vibrantvarun added 2 commits January 8, 2025 16:15

Fix error message of index.max_result_window setting

eae90dc

Signed-off-by: Varun Jain <[email protected]>

Rebase with main

7c586d8

Signed-off-by: Varun Jain <[email protected]>

vibrantvarun requested a review from martin-gaievski January 9, 2025 00:23

martin-gaievski approved these changes Jan 9, 2025

View reviewed changes

ryanbogan reviewed Jan 9, 2025

View reviewed changes

src/main/java/org/opensearch/neuralsearch/query/HybridQuery.java Show resolved Hide resolved

vibrantvarun requested review from ryanbogan January 9, 2025 21:27

ryanbogan approved these changes Jan 9, 2025

View reviewed changes

VijayanB reviewed Jan 9, 2025

View reviewed changes

src/main/java/org/opensearch/neuralsearch/query/HybridQueryBuilder.java Show resolved Hide resolved

VijayanB reviewed Jan 9, 2025

View reviewed changes

src/main/java/org/opensearch/neuralsearch/query/HybridQueryBuilder.java Outdated Show resolved Hide resolved

yuye-aws reviewed Jan 10, 2025

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

vibrantvarun added 2 commits January 10, 2025 08:39

Fixing validation condition for lower bound

5e04772

Signed-off-by: Varun Jain <[email protected]>

Merge branch 'main' into pagination

5b9a1bd

vibrantvarun requested review from VijayanB and yuye-aws January 10, 2025 16:40

martin-gaievski reviewed Jan 10, 2025

View reviewed changes

src/main/java/org/opensearch/neuralsearch/query/HybridQueryBuilder.java Outdated Show resolved Hide resolved

fix tests

33e328e

Signed-off-by: Varun Jain <[email protected]>

VijayanB approved these changes Jan 10, 2025

View reviewed changes

Removing version check from doEquals and doHashCode method

372ecbf

Signed-off-by: Varun Jain <[email protected]>

vibrantvarun merged commit b084838 into opensearch-project:main Jan 11, 2025
39 checks passed

vibrantvarun mentioned this pull request Jan 13, 2025

Test PR #1092

Closed

vibrantvarun mentioned this pull request Jan 14, 2025

[Backport 2.x] Pagination in hybrid query #1099

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pagination in Hybrid query #1048

Pagination in Hybrid query #1048

vibrantvarun commented Dec 31, 2024 •

edited

Loading

vibrantvarun commented Jan 2, 2025

martin-gaievski left a comment

vibrantvarun Jan 8, 2025

martin-gaievski left a comment

VijayanB left a comment

opensearch-trigger-bot bot commented Jan 11, 2025

Pagination in Hybrid query #1048

Pagination in Hybrid query #1048

Conversation

vibrantvarun commented Dec 31, 2024 • edited Loading

Description

Related Issues

Check List

vibrantvarun commented Jan 2, 2025

martin-gaievski left a comment

Choose a reason for hiding this comment

vibrantvarun Jan 8, 2025

Choose a reason for hiding this comment

martin-gaievski left a comment

Choose a reason for hiding this comment

VijayanB left a comment

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented Jan 11, 2025

vibrantvarun commented Dec 31, 2024 •

edited

Loading