Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-51073][SQL] Remove Unstable from SparkSessionExtensionsProvider trait #49780

Closed
wants to merge 1 commit into from
Closed

[SPARK-51073][SQL] Remove Unstable from SparkSessionExtensionsProvider trait #49780

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 1 addition & 4 deletions sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensionsProvider.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,10 @@

package org.apache.spark.sql

import org.apache.spark.annotation.{DeveloperApi, Since, Unstable}
import org.apache.spark.annotation.{DeveloperApi, Since}

// scalastyle:off line.size.limit
/**
* :: Unstable ::
*
* Base trait for implementations used by [[SparkSessionExtensions]]
*
*
Expand Down Expand Up @@ -76,7 +74,6 @@ import org.apache.spark.annotation.{DeveloperApi, Since, Unstable}
* @since 3.2.0
*/
@DeveloperApi
@Unstable
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh actually, I would prefer to have this as unstable because it can touch stuff like optimizer and analyzer rules that are internal API..

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand we won't likely remove SparkSessionExtensions, and this is service loader for SparkSessionExtensions (as mentioned in PR description). But I would prefer to avoid promoting this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you elaborate a little more about how SparkSessionExtensionsProvider are affected with a change of SparkSessionExtensions? IIUC, we are not going to remove or rename class SparkSessionExtensions .

trait SparkSessionExtensionsProvider extends Function1[SparkSessionExtensions, Unit]

ServiceLoader is only related to the relation between the text resource file and class name, SparkSessionExtensions, isn't it?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe, are you considering removing this ServiceLoader pattern, @HyukjinKwon ?

Copy link
Member

@HyukjinKwon HyukjinKwon Feb 4, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean, nothing will be affected by this ... I just worry that more people may SparkSessionExtensions itself...

TBH, I personally don't like this SparkSessionExtensions in general because it opens too much catalyst API usage, and touching rules.. and I was thinking about having them all as Unstable for now..

I actually don't like all those API ... such as df.queryExecution, etc. for the same reason.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am fine if others are okay - I don't have a strong opinion.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I understand your concerns fully. I also understand why this is in this status. However, Apache Spark depends on this since v2.2.0 (SPARK-18127) because this is the only way for certain features.

The goal of this PR is only this ServiceLoader pattern is applicable in the stable contract for the production usage (SRE or dev team) in Spark 4.1.0.

If many people are reluctant, I'm also going to drop this~ So, let's see.

@Since("3.2.0")
trait SparkSessionExtensionsProvider extends Function1[SparkSessionExtensions, Unit]
// scalastyle:on line.size.limit