[SPARK-51073][SQL] Remove Unstable from SparkSessionExtensionsProvider trait

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to remove Unstable from SparkSessionExtensionsProvider trait from Apache Spark 4.1.0.

spark/sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensionsProvider.scala

Lines 78 to 81 in 400e2b2

	@DeveloperApi
	@Unstable
	@Since("3.2.0")
	trait SparkSessionExtensionsProvider extends Function1[SparkSessionExtensions, Unit]

Why are the changes needed?

Since Apache Spark 3.2.0, we provide SparkSessionExtensionsProvider trait without any change to allow third-party developers to provide a resource file that contains default extensions.

• [SPARK-35380][SQL] Loading SparkSessionExtensions from ServiceLoader #32515

Apache Iceberg has been using this since 0.11.0.

• https://iceberg.apache.org/docs/1.7.1/spark-configuration/#sql-extensions

  Iceberg 0.11.0 and later add an extension module to Spark to add new SQL commands, like CALL for stored procedures or ALTER TABLE ... WRITE ORDERED BY.

Although SparkSessionExtensions is an unstable developer API, SparkSessionExtensionsProvider trait itself and ServiceLoader framework is stable and it's unlikely for Apache Spark to change this trait.

spark/sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensions.scala

Lines 106 to 109 in 400e2b2

	@DeveloperApi
	@Experimental
	@Unstable
	class SparkSessionExtensions {

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the CIs.

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

Hi, @yaooqinn . WDYT about this?

[HyukjinKwon]

Oh actually, I would prefer to have this as unstable because it can touch stuff like optimizer and analyzer rules that are internal API..

[HyukjinKwon]

I understand we won't likely remove SparkSessionExtensions, and this is service loader for SparkSessionExtensions (as mentioned in PR description). But I would prefer to avoid promoting this.

[dongjoon-hyun]

Could you elaborate a little more about how SparkSessionExtensionsProvider are affected with a change of SparkSessionExtensions? IIUC, we are not going to remove or rename class SparkSessionExtensions .

trait SparkSessionExtensionsProvider extends Function1[SparkSessionExtensions, Unit] 

ServiceLoader is only related to the relation between the text resource file and class name, SparkSessionExtensions, isn't it?

[dongjoon-hyun]

Maybe, are you considering removing this ServiceLoader pattern, @HyukjinKwon ?

[HyukjinKwon]

I mean, nothing will be affected by this ... I just worry that more people may SparkSessionExtensions itself...

TBH, I personally don't like this SparkSessionExtensions in general because it opens too much catalyst API usage, and touching rules.. and I was thinking about having them all as Unstable for now..

I actually don't like all those API ... such as df.queryExecution, etc. for the same reason.

[HyukjinKwon]

I am fine if others are okay - I don't have a strong opinion.

[dongjoon-hyun]

Yes, I understand your concerns fully. I also understand why this is in this status. However, Apache Spark depends on this since v2.2.0 (SPARK-18127) because this is the only way for certain features.

The goal of this PR is only this ServiceLoader pattern is applicable in the stable contract for the production usage (SRE or dev team) in Spark 4.1.0.

If many people are reluctant, I'm also going to drop this~ So, let's see.

[yaooqinn]

LGTM, the trait itself can be considered stable.

[dongjoon-hyun]

Thank you, @yaooqinn and @HyukjinKwon .

Merged to master for Apache Spark 4.1.0.