Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support object fields in star-tree index #16728

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

bharath-techie
Copy link
Contributor

@bharath-techie bharath-techie commented Nov 27, 2024

Description

We'll support object type fields with this change for star-tree index.
Star-tree index will still block array values for the object fields.

Example :

{
    "settings": {
        "index.number_of_shards": 1,
        "index.composite_index": true
    },
    "mappings": {
        "composite": {
            "startree1": {
                "type": "star_tree",
                "config": {
                    "ordered_dimensions": [
                        {
                            "name": "nested.status"
                        },
                        {
                            "name": "geoip.country_name"
                        }
                    ],
                    "metrics": [
                        {
                            "name": "nested.status"
                        }
                    ]
                }
            }
        },
        "properties": {
            "nested": {
                "properties" : {
                    "status": {
                        "type": "integer"
                    }
                }
            },
            "geoip": {
                "properties": {
                    "country_name": {
                        "type": "keyword"
                    }
                }
            }
        }
    }
}

Related Issues

#16730

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Copy link
Contributor

✅ Gradle check result for 1ae7561: SUCCESS

Copy link

codecov bot commented Nov 27, 2024

Codecov Report

Attention: Patch coverage is 75.00000% with 10 lines in your changes missing coverage. Please review.

Project coverage is 72.12%. Comparing base (b5f651f) to head (4da6d64).
Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
...va/org/opensearch/index/mapper/StarTreeMapper.java 72.72% 5 Missing and 1 partial ⚠️
...va/org/opensearch/index/mapper/DocumentParser.java 20.00% 1 Missing and 3 partials ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #16728      +/-   ##
============================================
- Coverage     72.21%   72.12%   -0.09%     
+ Complexity    65335    65315      -20     
============================================
  Files          5318     5318              
  Lines        304081   304119      +38     
  Branches      43995    44005      +10     
============================================
- Hits         219578   219360     -218     
- Misses        66541    66765     +224     
- Partials      17962    17994      +32     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@@ -690,6 +710,10 @@ public boolean isFieldPartOfCompositeIndex(String field) {
return fieldsPartOfCompositeMappings.contains(field);
}

public boolean isCompositeIndexFieldNestedField(String field) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a use-case where this method can be used separately from the isFieldPartOfCompositeIndex? Otherwise, can we combine both methods?

Copy link
Contributor

✅ Gradle check result for b1f8fba: SUCCESS

@sachinpkale sachinpkale changed the title Supporting object fields in star-tree index Support object fields in star-tree index Dec 19, 2024
@sachinpkale
Copy link
Member

Use imperative mood in the PR title - PR title becomes commit message after merging (I have changed it for this PR Supporting -> Support)

if (context.indexSettings().isCompositeIndex() && context.mapperService().isFieldPartOfCompositeIndex(arrayFieldName)) {
if (context.indexSettings().isCompositeIndex()
&& (context.mapperService().isFieldPartOfCompositeIndex(arrayFieldName)
|| context.mapperService().isFieldPartOfCompositeIndex(context.path().pathAsText(arrayFieldName)))) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have a test for this new check?

Copy link
Contributor Author

@bharath-techie bharath-techie Dec 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have tests for the new checks in IT. But I found an edge case where its failing.

{ "index" : { "_index" : "logs-2419985"  } }
{"@timestamp": 1706431663, "clientip": "215.79.163.253", 
"request": "GET /spanish/index.html HTTP/1.0", 
"nested": {"status": [218,219]}, "size": 202, "status": 218}

Say status is part of star-tree mapping but nested.status has array as above example.

arrayFieldName = "status"
context.path().pathAsText(arrayFieldName)) = "nested.status"

so the condition will satisfy for arrayFieldName and doc addition will be blocked.

@sachinpkale
Copy link
Member

Changes look good, please add changelog entry.

Copy link
Contributor

❌ Gradle check result for 995f912: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: bharath-techie <[email protected]>
Copy link
Contributor

✅ Gradle check result for 4da6d64: SUCCESS

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants