[SPARK-33538][SQL] Directly push IN/NOT predicates to the Hive Metastore

[yuexing]

What changes were proposed in this pull request?

This PR refactors the way IN and NOT IN predicates are converted in HiveShim. Previously, these predicates were expanded into chains of OR/AND comparisons (e.g., a = 1 or a = 2). This change updates the logic to use SQL-native IN and NOT IN syntax (e.g., a in (1, 2)), improving readability and possibly query performance.

Why are the changes needed?

Efficiency:
Many SQL engines optimize IN/NOT IN natively, which can improve performance and reduce query complexity.
Maintainability:
Simplifies predicate generation logic, making the codebase easier to maintain and extend.

Does this PR introduce any user-facing change?

No user-facing changes; this is an internal refactor that affects query generation logic.

How was this patch tested?

Existing unit tests for predicate conversion and query generation should be sufficient to cover the updated logic.

Was this patch authored or co-authored using generative AI tooling?

No.

[wangyum]

Suggested change

	def convertInDirectly(
	def convertIn(

[yuexing]

ok

[wangyum]

Revert this change

[yuexing]

done

[pan3793]

@yuexing do you have perf numbers for this change? with which kind/version of backend RDBMS of HMS?

[Tagar]

Blindly replacing = to IN (...) even for a single-element lookups can actually cause regressions, depending on HMS backend.

Worse performance: IN (...) can skip indexes or change query plans
probably affects pgsql, Oracle, SQL Server, MySQL etc. Needs to be properly tested for regressions!

NULL behavior differences: col = NULL vs col IN (NULL) behave differently.

Plan caching / cursor sharing changes - affects at least Oracle, SQL Server. May be a lesser issue, but could add up for metastores with high qps.

Safer Strategy - only rewrite to IN (...): If there's more than one value.

[Tagar]

Separately, spark.sql.hive.metastorePartitionPruningDirectInEnabled is too long? I would not understand what this means.
Perthaps spark.sql.hive.metastore.directInPruning or spark.sql.hive.hms.directIn is better? Just as an option

[yuexing]

Separately, spark.sql.hive.metastorePartitionPruningDirectInEnabled is too long? I would not understand what this means. Perthaps spark.sql.hive.metastore.directInPruning or spark.sql.hive.hms.directIn is better? Just as an option

hi @Tagar Thanks for commenting, let me reply to those questions;

1st, 'in' predicate with null is already skipped in old code, see

def convert(expr: Expression): Option[String] = expr match {
      case Not(InSet(_, values)) if values.size > inSetThreshold =>
        None

      case Not(In(_, list)) if hasNullLiteral(list) => None
      case Not(InSet(_, list)) if list.contains(null) => None
...

2nd, the conf is already removed following the code review comment. please see newest patch

[LuciferYang]

all test passed, friendly ping @wangyum

[LuciferYang]

+1, LGTM
@wangyum Do you need to take another look?

[wangyum]

Safer Strategy - only rewrite to IN (...): If there's more than one value.

@Tagar OptimizeIn will rewrite In to EqualTo If there's only one value:

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala

Lines 324 to 329 in da67217

	if (newList.length == 1
	// TODO: `EqualTo` for structural types are not working. Until SPARK-24443 is addressed,
	// TODO: we exclude them in this rule.
	&& !v.isInstanceOf[CreateNamedStruct]
	&& !newList.head.isInstanceOf[CreateNamedStruct]) {
	EqualTo(v, newList.head)

[LuciferYang]

Merged into master.Thanks @yuexing @wangyum @pan3793 and @Tagar