Update 2023-12-05-improving-document-retrieval-with-spade-semantic-en… #2483
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…coders.md Signed-off-by: Dagney <[email protected]>
dagneyb
requested review from
elfisher,
AMoo-Miki,
nknize,
krisfreedain,
peterzhuamazon,
CEHENKLE,
dtaivpp,
kolchfa-aws and
nateynateynate
as code owners
Signed-off-by: Fanit Kolchina <[email protected]>
Signed-off-by: Fanit Kolchina <[email protected]>
model-collapse
approved these changes
Dec 8, 2023
model-collapse
approved these changes
Dec 8, 2023
|
|
||
| In our previous [blog post](https://opensearch.org/blog/semantic-science-benchmarks), we mentioned that searching with dense embeddings presents challenges when encoders are facing unfamiliar content. When an encoder trained on one dataset is used on a different dataset, the encoder often produces unpredictable embeddings, resulting in poor search result relevance. | ||
|
|
||
| Often, BM25 performs better than dense encoders on BEIR datasets that incorporate strong domain knowledge. In these cases, sparse encoders fall back on keyword-based matching, ensuring that their search results are no worse than BM25 ones. For a comparison of search result relevance benchmarks, see [Table I](#benchmarking-results). |
There was a problem hiding this comment.
on some BEIR dataset which incorporate
There was a problem hiding this comment.
sparse encoders are able to fall back on keyword-based ...
|
|
||
| You can run a neural sparse search in two modes: **bi-encoder** and **document-only**. | ||
|
|
||
| In bi-encoder mode, both documents and search queries are passed through deep encoders. In document-only mode, documents are still passed through deep encoders, but search queries are instead tokenized. In this mode, document encoders are trained to learn more synonym association in order to increase recall. By eliminating the online inference phase, you can **save computational resources** and **significantly reduce latency**. For benchmarks, compare the `Neural sparse doc-only` column with other columns in [Table II](#table-ii-speed-comparison-in-terms-of-latency-and-throughput). |
There was a problem hiding this comment.
search queries are just simply tokenized instead.
natebower
reviewed
Dec 8, 2023
There was a problem hiding this comment.
@kolchfa-aws Please see my comments and changes and let me know if you have any questions. Thanks!
_posts/2023-12-05-improving-document-retrieval-with-sparse-semantic-encoders.md
Outdated
Show resolved
Hide resolved
| has_science_table: true | ||
| --- | ||
|
|
||
| OpenSearch 2.11 introduced neural sparse search---a new, efficient way of semantic retrieval. In this blog post, you'll learn about using sparse encoders for semantic search. You'll find that neural sparse search reduces costs, performs faster, and improves search relevance. We're excited to share benchmarking results that show why neural sparse search is now the top-performing search method. You can even try it out by building your own search engine in just five steps. For a TLDR on benchmarking learnings, see [Key takeaways](#here-are-the-key-takeaways). |
There was a problem hiding this comment.
Let's replace TLDR as too informal. In the fourth sentence, let's clarify slightly. The top-performing search method as compared to what?
|
|
||
| When you use a transformer-based encoder, such as BERT, to generate traditional dense vector embeddings, the encoder translates each word into a vector. Collectively, these vectors make up a semantic vector space. In this space, the closer the vectors are, the more similar the words are in meaning. | ||
|
|
||
| In sparse encoding, the encoder takes the text and creates a list of tokens that have similar semantic meaning. The model vocabulary ([WordPiece](https://huggingface.co/learn/nlp-course/chapter6/6?fw=pt)) contains most commonly used words along with various tense endings (for example, `-ed` and `-ing`) and suffixes (for example, `-ate` and `-ion`). You can think of the vocabulary as a semantic space where each document is a sparse vector. |
There was a problem hiding this comment.
First sentence: "uses the text to create"?
|
|
||
| ## Sparse encoders perform better on unfamiliar datasets | ||
|
|
||
| In our previous [blog post](https://opensearch.org/blog/semantic-science-benchmarks), we mentioned that searching with dense embeddings presents challenges when encoders are facing unfamiliar content. When an encoder trained on one dataset is used on a different dataset, the encoder often produces unpredictable embeddings, resulting in poor search result relevance. |
There was a problem hiding this comment.
"encounter" instead of "are facing" (or a better verb)?
|
|
||
| In our previous [blog post](https://opensearch.org/blog/semantic-science-benchmarks), we mentioned that searching with dense embeddings presents challenges when encoders are facing unfamiliar content. When an encoder trained on one dataset is used on a different dataset, the encoder often produces unpredictable embeddings, resulting in poor search result relevance. | ||
|
|
||
| Often, BM25 performs better than dense encoders on BEIR datasets that incorporate strong domain knowledge. In these cases, sparse encoders fall back on keyword-based matching, ensuring that their search results are no worse than BM25 ones. For a comparison of search result relevance benchmarks, see [Table I](#benchmarking-results). |
There was a problem hiding this comment.
Instead of "than BM25 ones", "are no worse than those produced by BM25"?
| The `neural_sparse` query supports two parameters: | ||
|
|
||
| - `model_id` (String): The ID of the model that is used to generate tokens and weights from the query text. A sparse encoding model will expand the tokens from query text, while the tokenizer model will only tokenize the query text itself. | ||
| - `max_token_score` (Float): An extra parameter required for performance optimization. Just like the OpenSearch `match` query, the `neural_sparse` query is transformed to a Lucene BooleanQuery, combining term-level subqueries using disjunction. The difference is that for the neural sparse query, we use FeatureQuery instead of TermQuery to match the terms. Lucene employs the WAND (Weak AND) algorithm for dynamic pruning, which skips non-competitive tokens based on their score upper bounds. However, FeatureQuery uses `FLOAT.MAX_VALUE` as the score upper bound, which makes the WAND optimization ineffective. The `max_token_score` parameter resets the score upper bound for each token in a query, which is consistent with the original FeatureQuery. Thus, setting the value to 3.5 for the bi-encoder model and 2 for the document-only model can accelerate search without precision loss. After OpenSearch is upgraded to Lucene version 9.8, this parameter will be deprecated. |
_posts/2023-12-05-improving-document-retrieval-with-sparse-semantic-encoders.md
Outdated
Show resolved
Hide resolved
_posts/2023-12-05-improving-document-retrieval-with-sparse-semantic-encoders.md
Outdated
Show resolved
Hide resolved
| ## Next steps | ||
|
|
||
| - For more information about neural sparse search, see [Neural sparse search](https://opensearch.org/docs/latest/search-plugins/neural-sparse-search/). | ||
| - For an OpenSearch end-to-end neural search tutorial, see [Neural search tutorial](https://opensearch.org/docs/latest/search-plugins/neural-search-tutorial/). |
There was a problem hiding this comment.
Should "OpenSearch" follow "end-to-end"?
_posts/2023-12-05-improving-document-retrieval-with-sparse-semantic-encoders.md
Outdated
Show resolved
Hide resolved
kolchfa-aws
reviewed
Dec 8, 2023
_posts/2023-12-05-improving-document-retrieval-with-sparse-semantic-encoders.md
Outdated
Show resolved
Hide resolved
kolchfa-aws
reviewed
Dec 8, 2023
_posts/2023-12-05-improving-document-retrieval-with-sparse-semantic-encoders.md
Outdated
Show resolved
Hide resolved
Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]>
Signed-off-by: Fanit Kolchina <[email protected]>
Signed-off-by: Fanit Kolchina <[email protected]>
Signed-off-by: Fanit Kolchina <[email protected]>
Signed-off-by: Fanit Kolchina <[email protected]>
Signed-off-by: Fanit Kolchina <[email protected]>
natebower
approved these changes
Dec 8, 2023
There was a problem hiding this comment.
LGTM. @pajuric This should be ready to republish whenever you're ready. @kolchfa-aws can merge for you.
Signed-off-by: Fanit Kolchina <[email protected]>
|
@pajuric Let me know if I should adjust the meta/keywords. |
Signed-off-by: Fanit Kolchina <[email protected]>
Signed-off-by: Fanit Kolchina <[email protected]>
natebower
reviewed
Dec 8, 2023
Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]>
Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]>
Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]>
Signed-off-by: Fanit Kolchina <[email protected]>
…into dagney-branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…coders.md
Description
[Describe what this change achieves]
Issues Resolved
[List any issues this PR will resolve]
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the BSD-3-Clause License.