[MOD-10184] add some optimizations to simplify some patterns in iterators

[tonyd3]

Iterator Tree Simplification and Query Parser Optimizations

This pull request introduces significant optimizations to the query iterator tree construction and the query parser. The primary goal is to simplify complex iterator combinations into more efficient forms (e.g., replacing an intersection with an empty child with an empty iterator) and to optimize query parsing by eliminating redundant negations. These changes aim to improve query performance and reduce memory overhead by creating more streamlined iterator execution plans.

Key Changes

• Implemented iterator reduction rules for IntersectionIterator, UnionIterator, NotIterator, OptionalIterator, and HybridVectorIterator to simplify the iterator tree based on child types (e.g., empty, null, wildcard).
• Introduced a IsWildcardIterator helper function and an isWildcard flag within InvIndIterator to correctly identify iterators that behave like wildcards, enabling more aggressive optimizations.
• Optimized both v1 and v2 query parsers to eliminate double negations (e.g., ```NOT(NOT``(A))becomesA`), improving query tree efficiency.
• Added a ``notSubtree`` flag to ``QueryEvalCtx`` to influence the behavior of ``UnionIterator`` when it is part of a ``NOT`` subtree, allowing for `quick exit` optimizations.
• Updated ``NewWildcardIterator_NonOptimized`` and ``NewInvIndIterator_GenericQuery`` to accept a `weight` parameter, ensuring consistent weight propagation.
• Extensively updated C++ unit tests across multiple iterator types to validate the new reduction logic and ensure correct memory management and behavior in various edge cases.

Affected Areas

• src/iterators/ (intersection_iterator.c, union_iterator.c, not_iterator.c, optional_iterator.c, wildcard_iterator.c, inverted_index_iterator.c, inverted_index_iterator.h, wildcard_iterator.h)
• src/query_parser/ (v1/parser.c, v1/parser.y, v2/parser.c, v2/parser.y)
• src/query.c
• src/hybrid_reader.c
• src/query_ctx.h
• tests/cpptests/ (test_cpp_iterator_intersection.cpp, test_cpp_iterator_union.cpp, test_cpp_iterator_not.cpp, test_cpp_iterator_optional.cpp, test_cpp_index.cpp, test_cpp_query.cpp, test_cpp_iterator_index.cpp, micro-benchmarks/benchmark_wildcard_non_optimized_iterator.cpp, micro-benchmarks/benchmark_index_iterator.cpp)
• tests/pytests/ (test_issues.py, test_profile.py)

---

This summary was automatically generated by @propel-code-bot

[propel-code-bot]

[BestPractice]

This check for WILDCARD_ITERATOR does not handle optimized wildcard iterators, which are of type READ_ITERATOR with a special flag. This can lead to missed optimizations for hybrid queries.

You've introduced IsWildcardIterator() to solve this, and it's used in the other new reducer functions. This reducer should also use IsWildcardIterator(hParams->childIt) to correctly handle all types of wildcard iterators. You'll also need to include iterators/wildcard_iterator.h at the top of the file.

Suggested change

	} else if (hParams->childIt && hParams->childIt->type == WILDCARD_ITERATOR) {
	//TODO: When new Iterator API (consider READER_ITERATOR with isWildcard flag)
	hParams->childIt->Free(hParams->childIt);
	hParams->qParams.searchMode = VECSIM_STANDARD_KNN;
	hParams->childIt = NULL;
	}
	} else if (hParams->childIt && IsWildcardIterator(hParams->childIt)) {
	//TODO: When new Iterator API (consider READER_ITERATOR with isWildcard flag)
	hParams->childIt->Free(hParams->childIt);
	hParams->qParams.searchMode = VECSIM_STANDARD_KNN;
	hParams->childIt = NULL;
	}

⚡ Committable suggestion

Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.

[propel-code-bot]

[TestCoverage]

The comment here indicates that with the new optimizations, the expected result should be an EMPTY iterator profile, because an intersection with a non-existent term (new) should be reduced to an empty set. The new IntersectionIteratorReducer implements this logic.

However, the test still asserts the old, unoptimized behavior. Please update this test (and a similar one in test_profile.py) to assert the new, correct, optimized behavior. This will ensure the optimizations are working as intended and prevent future regressions.

[propel-code-bot]

[TestCoverage]

The comment here indicates that with the new optimizations, the NOT iterator should be reduced to a WILDCARD iterator because its child is an EMPTY iterator (-@t:baz where baz doesn't exist). The new NotIteratorReducer implements this logic.

However, the test still asserts the old, unoptimized behavior. Please update this test to assert the new, correct, optimized behavior to ensure the optimizations are working as intended.