Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Indices that include knn_vector field fail when search includes "fields" parameter #1633

Open
jmazanec15 opened this issue Apr 19, 2024 · 7 comments · May be fixed by #2314
Open

[BUG] Indices that include knn_vector field fail when search includes "fields" parameter #1633

jmazanec15 opened this issue Apr 19, 2024 · 7 comments · May be fixed by #2314
Assignees
@e-emoto
Labels
backlog bug Something isn't working good first issue Good for newcomers

Comments

@jmazanec15
Copy link
Member

Description

Right now, for indices that contain a knn_vector, we are not able to support the fields option in queries. This is because in KNNVectorFieldType, we do not implement ValuesFetcher. We should implement it similar to the geo types that handle arrays directly.

curl -X GET "localhost:9200/target_index/_search?pretty" -H 'Content-Type: application/json' -d'
> {
>   "fields": ["*"],
>   "query": {
>     "match_all": {}
>   }
> }
> '

{
  "error" : {
    "root_cause" : [
      {
        "type" : "unsupported_operation_exception",
        "reason" : "KNN Vector do not support fields search"
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "target_index",
        "node" : "grr7rDoPSwKXqQcpNrcbJQ",
        "reason" : {
          "type" : "unsupported_operation_exception",
          "reason" : "KNN Vector do not support fields search"
        }
      }
    ],
    "caused_by" : {
      "type" : "unsupported_operation_exception",
      "reason" : "KNN Vector do not support fields search",
      "caused_by" : {
        "type" : "unsupported_operation_exception",
        "reason" : "KNN Vector do not support fields search"
      }
    }
  },
  "status" : 500
}

Repro steps

curl -X PUT "localhost:9200/target_index" -H 'Content-Type: application/json' -d'
{
  "settings" : {
    "number_of_shards" : 1,
    "number_of_replicas" : 0,
    "index.knn": true
  },
  "mappings": {
       "properties": {
       "target_field": {
           "type": "knn_vector",
           "dimension": 2,
           "method": {
               "name":"hnsw",
               "engine":"nmslib",
               "space_type": "l2",
               "parameters":{
                 "m":20,
                 "ef_construction": 23
               }
           }
      }
   }
  }
}
'

curl -X PUT "localhost:9200/_bulk" -H 'Content-Type: application/json' -d'
{ "index": { "_index": "target_index" } }
{ "target_field": [1.5, 5.5]}
{ "index": { "_index": "target_index" } }
{ "target_field": [2.5, 3.5]}
{ "index": { "_index": "target_index" } }
{ "target_field": [4.5, 5.5]}
{ "index": { "_index": "target_index" } }
{ "target_field": [1.5, 5.5]}
{ "index": { "_index": "target_index" } }
{ "target_field": [1.5, 5.5]}
{ "index": { "_index": "target_index" } }
{ "target_field": [2.5, 3.5]}
'


curl localhost:9200/target_index/_refresh?pretty
curl -X GET "localhost:9200/target_index/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "fields": ["*"],
  "query": {
    "match_all": {}
  }
}
'
@jmazanec15 jmazanec15 added the bug Something isn't working label Apr 19, 2024
@OliverLiebmann
Copy link

This is not a bug, you are trying to use efficient knn filtering, but used the nmslib engine, which does not support it. If you change the engine to lucene or faiss it should work as expected.

Here is the documentation for it: https://opensearch.org/docs/latest/search-plugins/knn/filter-search-knn/
image

@jmazanec15
Copy link
Member Author

@OliverLiebmann this is a bug - it doesnt have anything to do with filtering - more so use of fields in the search payload. See:

curl -X GET "localhost:9200/target_index/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "fields": ["*"],
  "query": {
    "match_all": {}
  }
}
'

@jmazanec15
Copy link
Member Author

See this comment as well: https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/index/mapper/MappedFieldType.java#L128-L134

@sam-herman
Copy link

@jmazanec15 feel free to assign to me I understand the required fix.

@jmazanec15
Copy link
Member Author

Thanks @samuel-oci ! Assigned

@jmazanec15
Copy link
Member Author

@samuel-oci are you taking a look at this one?

@jmazanec15 jmazanec15 added the good first issue Good for newcomers label Oct 16, 2024
@jmazanec15 jmazanec15 moved this from Backlog to Backlog (Hot) in Vector Search RoadMap Oct 16, 2024
@jmazanec15 jmazanec15 assigned vamshin and unassigned sam-herman Oct 16, 2024
@e-emoto
Copy link

e-emoto commented Nov 25, 2024

I will start working on this task

@e-emoto e-emoto linked a pull request Dec 7, 2024 that will close this issue
Open
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@e-emoto e-emoto

Labels
backlog bug Something isn't working good first issue Good for newcomers
Projects
Vector Search RoadMap
Status: Backlog (Hot)
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add support for search using the "fields" parameter with knn_vector field e-emoto/k-NN
6 participants
@yigithub @OliverLiebmann @jmazanec15 @vamshin @e-emoto @sam-herman