DISC-378 : Create persona filters on hierachical qualifiedName filters

[sumandas0]

Hierarchical filters on qualifiedName to eliminate the wildcard persona filters.

We are creating an internal attributes in ES which will store the hierarchical data of qualifiedName which we will use to filter assets based on hierarchy and get rid of wildcard filters in most of the cases. More details is available on https://www.notion.so/atlanhq/Problem-Statement-Optimizing-Persona-Aliases-in-Elasticsearch-for-Improved-Search-Performance-7840c233e6dd4d96b017dff791c88282?pvs=4

[nikhilbonte21]

For asset value default/redshift/1705171197/dev/public/*
isHierarchical will be true
isWildcard will be true

Given this, it will never go to else block & hence metadataPolicyQualifiedNames.add(asset); will not be executed instead it will create wildcard as isWildcard is true

Ideally in this case asset should be added into metadataPolicyQualifiedNames (substring by -2) & no need of wildcard then

[nikhilbonte21]

Also, honestly, I feel this logic is complex to understand & maintain, and different check for useHierarchicalQualifiedNameFilter

can we separate them by useHierarchicalQualifiedNameFilter?

like

boolean isWildcard = asset.contains("*") || asset.contains("?");

if (useHierarchicalQualifiedNameFilter) {
    if (isWildcard) {
        boolean isHierarchical = asset.endsWith("/*");

        if (isHierarchical) {
	    asset = asset.substring(0, asset.length() - 2);
	    metadataPolicyQualifiedNames.add(asset);
	} else {
	    allowClauseList.add(mapOf("wildcard", mapOf(QUALIFIED_NAME, asset)));
	    allowClauseList.add(mapOf("wildcard", mapOf(QUALIFIED_NAME, asset + "/*")));
	}
    } else {
    	metadataPolicyQualifiedNames.add(asset);
    }

} else {
    // this is as it is previous logic
    if (isWildcard) {
        allowClauseList.add(mapOf("wildcard", mapOf(QUALIFIED_NAME, asset)));
    } else {
        terms.add(asset);
    }
    allowClauseList.add(mapOf("wildcard", mapOf(QUALIFIED_NAME, asset + "/*")));
}

[sumandas0]

For asset value default/redshift/1705171197/dev/public/*
isHierarchical will be true
isWildcard will be true

No asset = asset.substring(0, asset.length() - 2); will make sure we we cut /* from the string and consider it then. So isWildcard will be false.

[sumandas0]

This code is like this because if we wanted to optimize for the case of /* in the end. If there is any we trim last part and proceed.

And added extra 2 conditions if we have hierarchical flag enabled, when we enable it for everyone and remove flag condition the conditions will be simpler and easier to write code then. Its bit messy for the feature flag part actually. Otherwise its pretty simple.

Its already been tested a couple of time, I think we can proceed with this one.

[nikhilbonte21]

Understood

[nikhilbonte21]

Minor: call this statement only if necessary as it is redundant in case of GCTs, Readme, Link

[nikhilbonte21]

Minor suggestion:
This If can be avoided by moving currentPath.append("/"); to the very end of the block (inside)
if (StringUtils.isNotEmpty(part)) {

[nikhilbonte21]

e.g.

if (StringUtils.isNotEmpty(part))
    currentPath.append(part);
    // i>1 reason: we don't want to add the first part of the qualifiedName as it is the entity name
    // Example qualifiedName : default/snowflake/123/db_name we only want `default/snowflake/123` and `default/snowflake/123/db_name`
    if (i > qualifiedNameOffset) {
        if (isDataMeshType && (part.equals("domain") || part.equals("product"))) {
            continue;
        }
        AtlasGraphUtilsV2.addEncodedProperty(vertex, QUALIFIED_NAME_HIERARCHY_PROPERTY_KEY, currentPath.toString());
    }
    currentPath.append("/");
}

[sumandas0]

Its there if there is no / in the qName then the split would be of len 1, and we will basically just not add / which will not be handled in this case. This condition makes sure it doesn't add unnecessary / after qualifiedName of glossary like assets.

[nikhilbonte21]

All we are doing is i > 0 before currentPath.append(part);
Above suggestion results into exact same output & avoids i>0 check many times in case QN is longer & with many /, but anyway this is not a blocker to merge