|
| 1 | +--- |
| 2 | +layout: default |
| 3 | +title: Vector search with MMR |
| 4 | +nav_order: 60 |
| 5 | +parent: Specialized vector search |
| 6 | +has_children: false |
| 7 | +has_math: true |
| 8 | +--- |
| 9 | + |
| 10 | +# Vector search with MMR |
| 11 | + |
| 12 | +The maximal marginal relevance (MMR) search helps balance relevance and diversity in search results. Instead of returning only the most similar documents, MMR selects results that are both relevant to the query and different from each other. This improves the coverage of the result set and reduces redundancy, which is especially useful in vector search scenarios. |
| 13 | + |
| 14 | +MMR re-ranking balances two competing objectives: |
| 15 | + |
| 16 | + - Relevance: How well a document matches the query. |
| 17 | + |
| 18 | + - Diversity: How different a document is from the documents already selected. |
| 19 | + |
| 20 | +The algorithm computes a score for each candidate document using the following principle: |
| 21 | + |
| 22 | +```json |
| 23 | +MMR = (1 − λ) * relevance_score − λ * max(similarity_with_selected_docs) |
| 24 | +``` |
| 25 | + |
| 26 | +Where: |
| 27 | + |
| 28 | + - λ is the diversity parameter (closer to 1 means higher diversity). |
| 29 | + |
| 30 | + - relevance_score measures similarity between the query vector and the candidate document vector. |
| 31 | + |
| 32 | + - similarity_with_selected_docs measures similarity between the candidate and already selected documents. |
| 33 | + |
| 34 | +By adjusting the diversity parameter, you can control the tradeoff between highly relevant results and more diverse coverage in the result set. |
| 35 | + |
| 36 | +# Prerequisites |
| 37 | + |
| 38 | +To use MMR, you must enable [system-generated search processor factories](({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/system-generated-search-processors/)). Set the `cluster.search.enabled_system_generated_factories` setting (by default it is an empty list) to either `*` or explicitly include the required factories: |
| 39 | + |
| 40 | +```json |
| 41 | +PUT _cluster/settings |
| 42 | +{ |
| 43 | + "persistent": { |
| 44 | + "cluster.search.enabled_system_generated_factories": [ |
| 45 | + "mmr_over_sample_factory", |
| 46 | + "mmr_rerank_factory" |
| 47 | + ] |
| 48 | + } |
| 49 | +} |
| 50 | +``` |
| 51 | +{% include copy-curl.html %} |
| 52 | + |
| 53 | +# Parameters |
| 54 | + |
| 55 | +The mmr extension in the search API supports the following parameters: |
| 56 | + |
| 57 | +| Parameter | Data type | Required | Description | |
| 58 | +| ------------------------- | --------- | ----------------------------------------- |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |
| 59 | +| `diversity` | float | No | Controls the weight of diversity in the re-ranking process. Valid values range from `0` to `1`. A value of `1` prioritizes maximum diversity, and `0` disables diversity. Default is `0.5`. | |
| 60 | +| `candidates` | integer | No | Specifies how many candidate documents to oversample before re-ranking. Default is `3 * query size`. | |
| 61 | +| `vector_field_path` | string | Optional, but required for remote indices | Path to the vector field used for MMR re-ranking. If not provided, OpenSearch resolves it automatically from the search request. | |
| 62 | +| `vector_field_data_type` | string | Optional, but required for remote indices | Data type of the vector field. Used to parse the field and calculate similarity. If not provided, OpenSearch resolves it from the index mapping. | |
| 63 | +| `vector_field_space_type` | string | Optional, but required for remote indices | Used to decide the similarity function for the vector field, such as cosine similarity or Euclidean distance. If not provided, OpenSearch resolves it from the index mapping. | |
| 64 | + |
| 65 | + |
| 66 | +# Example request |
| 67 | + |
| 68 | +The following example shows how to use the mmr extension with a k-NN query: |
| 69 | + |
| 70 | +```json |
| 71 | +POST /my-index/_search |
| 72 | +{ |
| 73 | + "query": { |
| 74 | + "knn": { |
| 75 | + "my_vector_field": { |
| 76 | + "vector": [0.12, 0.54, 0.91], |
| 77 | + "k": 10 |
| 78 | + } |
| 79 | + } |
| 80 | + }, |
| 81 | + "ext": { |
| 82 | + "mmr": { |
| 83 | + "diversity": 0.7 |
| 84 | + } |
| 85 | + } |
| 86 | +} |
| 87 | + |
| 88 | +``` |
| 89 | +{% include copy-curl.html %} |
| 90 | + |
| 91 | +The following example shows how to use the mmr extension with a neural query: |
| 92 | +```json |
| 93 | +POST /my-index/_search |
| 94 | +{ |
| 95 | + "query": { |
| 96 | + "neural": { |
| 97 | + "my_vector_field": { |
| 98 | + "query_text": "query text", |
| 99 | + "model_id": "<your model id>" |
| 100 | + } |
| 101 | + } |
| 102 | + }, |
| 103 | + "ext": { |
| 104 | + "mmr": { |
| 105 | + "diversity": 0.6, |
| 106 | + "candidates": 50, |
| 107 | + "vector_field_path": "my_vector_field", |
| 108 | + "vector_field_data_type": "float", |
| 109 | + "vector_field_space_type": "l2" |
| 110 | + } |
| 111 | + } |
| 112 | +} |
| 113 | +``` |
| 114 | + |
| 115 | +When querying across multiple indices, ensure that the data type, and space type are aligned. Since that info decides the similarity function we use to calculate the similarity between docs. |
| 116 | +{: .note} |
| 117 | + |
| 118 | +# Limitations |
| 119 | + |
| 120 | +## MMR Query Type Restriction: |
| 121 | +MMR currently only supports knn or neural queries as the top-level query in a search request. If knn or neural is nested inside another query type (such as a bool query or hybrid query), MMR is not supported. |
| 122 | + |
| 123 | +## Required Explicit Vector Field Details |
| 124 | +You must explicitly provide the vector field details—`vector_field_path, vector_field_data_type, and vector_field_space_type`—when querying remote indices. |
| 125 | + |
| 126 | +Reason: Unlike a local index where OpenSearch can automatically resolve this metadata from the index mapping, the system cannot reliably fetch this information from the remote cluster. Providing these details ensures correct parsing of the vector data and accurate similarity calculations. |
0 commit comments