[BUG] term_vector does not work on search_as_you_type fields

[haversnail]

Describe the bug
TL;DR: same issue reported here.

Per ElasticSearch's 7.10 documentation (which AFAICT should be compatible with OpenSearch 1.x), setting term_vector on a search_as_you_type field should only affect the shingle subfields, but not the index prefix subfield. However, when attempting to index documents with this mapping, an illegal_argument_exception is thrown.

To Reproduce
Steps to reproduce the behavior:

1. Create a new index with the following mapping:

{
  // ...
  "my_field":
    "type": "search_as_you_type",
    "term_vector": "with_positions_offsets"
  // ...
}

2. Attempt to index a document containing a my_field field
3. Observe the following error:

{
  "type": "illegal_argument_exception",
  "reason": "cannot index term vector offsets when term vectors are not indexed (field=\"my_field._index_prefix\")"
}

Expected behavior
Documents should be indexed without issue. Only the ._2gram and ._3gram shingle subfields should have the term_vector set, and not the ._index_prefix n-gram subfield.

Plugins
N/A

Screenshots
N/A

Host/Environment

• Version: 1.2.3 (opensearchproject/opensearch:latest)

[haversnail]

I noticed there was a PR that claims to resolve the issue, however I just tried this with the latest version of OpenSearch and am still encountering the same issue. ☹️

type: 'illegal_argument_exception',
reason: 'cannot index term vector offsets when term vectors are not indexed (field="name._index_prefix")'

[dblock]

@vpehkone ^

[vpehkone]

I tested with below OpenSearch docker versions and single node cluster. Didn't get errors after 2.0.0 version.

2.7.0: no errors
2.6.0: no errors
2.5.0: no errors
2.4.0: no errors
2.3.0: no errors
2.2.0: no errors
2.1.0: no errors
2.0.0: got the error

Test commands:

curl -XPUT -H "Content-type: application/json" http://localhost:9200/myindex?pretty -d '{ "mappings": { "properties": { "myfield": { "type": "search_as_you_type", "term_vector": "with_positions_offsets" }}}}'

curl -XPOST -H "Content-type: application/json" http://localhost:9200/myindex/_doc/1?pretty -d '{"myfield":"XXXX"}'

Responses if no error:

{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "myindex"
}

{
  "_index" : "myindex",
  "_id" : "1",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 0,
  "_primary_term" : 1
}

Responses if error:

{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "myindex"
}

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "cannot index term vector offsets when term vectors are not indexed (field=\"myfield._index_prefix\")"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "cannot index term vector offsets when term vectors are not indexed (field=\"myfield._index_prefix\")"
  },
  "status" : 400
}

[dblock]

@haversnail which version have you tried? any different from what @vpehkone is doing?

it would be great if someone could turn this into a REST YAML test if we don't already have one

[sejli]

@YashAnetz will be trying this one out for OSCI. Great starting point would be the YAML test that dblock@ mentioned

[msfroh]

I could be mistaken, but I think a fix similar to the one from #12048 might help here.

Specifically, I think the issue is that it's trying to create a query on one of these "internal" fields. It should probably add a check to see if the field name contains a dot (.), and if so, is the character following the last dot an underscore(_). If so, it's an internal field and should be skipped.

[getsaurabh02]

We discussed this in the Search community meeting today. Eric Pugh volunteered to take a stab at it.

[epugh]

I've got a build running on my Mac now, and am more comfortable pickign this one up!

[sandeshkr419]

@epugh Did you get a chance to work on this further?