Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] OpenSearch should create _seq_no field mapping at Index creation itself and NOT after atleast 1 doc is ingested #16185

Open
eirsep opened this issue Oct 3, 2024 · 2 comments
Labels
bug Something isn't working Indexing Indexing, Bulk Indexing and anything related to indexing

Comments

@eirsep
Copy link
Member

eirsep commented Oct 3, 2024

Describe the bug

_seq_no field is not created by OpenSearch at index creation time. It seems to be created at the time of ingestion of 1 st document for that index

When our plugin (Security Analytics) fetches data from an index, it tries to sort by _seq_no which fails on empty index instead of returning empty result.

Related component

Indexing

To Reproduce

  1. Create index
  2. Before ingesting data, search on index with sort field as _seq_no
    3.Error shows up
Caused by: org.opensearch.index.query.QueryShardException: No mapping found for [_seq_no] in order to sort on
        at org.opensearch.search.sort.FieldSortBuilder.resolveUnmappedType(FieldSortBuilder.java:564) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.search.sort.FieldSortBuilder.build(FieldSortBuilder.java:411) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.search.sort.SortBuilder.buildSort(SortBuilder.java:168) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.search.SearchService.parseSource(SearchService.java:1268) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.search.SearchService.createContext(SearchService.java:998) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.search.SearchService.executeQueryPhase(SearchService.java:606) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:579) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:74) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]


### Expected behavior

OpenSearch should use a protected index template that defines metadata mappings like _version, _seq_no for all indices at index createion time itself

### Additional Details

_No response_
@eirsep eirsep added bug Something isn't working untriaged labels Oct 3, 2024
@github-actions github-actions bot added the Indexing Indexing, Bulk Indexing and anything related to indexing label Oct 3, 2024
@bharath-techie
Copy link
Contributor

When our plugin (Security Analytics) fetches data from an index, it tries to sort by _seq_no which fails on empty index instead of returning empty result.

@eirsep can't this be handled in plugin ? any specific use case on why we do this query ?

@bharath-techie
Copy link
Contributor

[Triage attendees - 1 2 3 4]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working Indexing Indexing, Bulk Indexing and anything related to indexing
Projects
None yet
Development

No branches or pull requests

2 participants