[KYUUBI #5336] Spark extension supports Spark 3.5 #5336
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
kind:infra
wForget
changed the title
wForget
commented
Sep 27, 2023
...uubi-extension-spark-3-5/src/main/scala/org/apache/kyuubi/sql/KyuubiEnsureRequirements.scala
Outdated
Show resolved
Hide resolved
wForget
changed the title
Codecov Report
@@ Coverage Diff @@
## master #5336 +/- ##
======================================
Coverage 0.00% 0.00%
======================================
Files 590 588 -2
Lines 33493 33399 -94
Branches 4424 4387 -37
======================================
+ Misses 33493 33399 -94 see 2 files with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
|
Seems CI does not cover Spark 3.5 |
wForget
changed the title
I have added CI for spark 3.5 and it seems that there are some failed tests, I will try to fix them. |
wForget
commented
Oct 7, 2023
| @@ -157,7 +157,7 @@ | |||
| }, { | |||
| "classname" : "org.apache.spark.sql.catalyst.plans.logical.CreateTableAsSelect", | |||
| "tableDescs" : [ { | |||
| "fieldName" : "left", | |||
| "fieldName" : "name", | |||
There was a problem hiding this comment.
There is no left method in spark 3.5.
spark 3.4 related code:
https://github.com/apache/spark/blob/e056fc8fb4d1c1ab8786a0989adc7298cc918dae/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala#L444
spark 3.5 related code:
https://github.com/apache/spark/blob/64e2b22f6b4023197871a60eb08b055688e9fdd2/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala#L467
wForget
changed the title
|
@pan3793 @ulysses-you The test cases have been run successfully and this is ready for review. cc @bowenliang123 |
pan3793
reviewed
Oct 7, 2023
pan3793
reviewed
Oct 7, 2023
pan3793
changed the title
16 tasks
ulysses-you
approved these changes
Oct 8, 2023
pan3793
changed the title
pan3793
pushed a commit
that referenced
this pull request
Oct 9, 2023
### _Why are the changes needed?_ It is basically copied from `kyuubi-extension-spark-3-4`. ### _How was this patch tested?_ Compiled successfully: ``` build/mvn clean install -DskipTests -Pflink-provided,spark-provided,hive-provided,spark-3.5 ``` - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes #5336 from wForget/dev_spark_3_5. Closes #5336 7ba9980 [wforget] remove iceberg.version in spark-3.5 profile a18ce16 [wforget] Regenerate KyuubiEnsureRequirements based on EnsureRequirements in spark 3.5 4725c47 [wforget] fix iceberg version f5a8ea9 [wforget] Bump iceberg 1.4.0 06558dc [wforget] make kyuubi-spark-authz plugin compatible with Spark3.5 90d0e4c [wforget] make kyuubi-spark-authz plugin compatible with Spark3.5 4bc8d24 [wforget] add ci 1b3f2d9 [wforget] Make kyuubi spark extension compatible with Spark3.5 Authored-by: wforget <[email protected]> Signed-off-by: Cheng Pan <[email protected]> (cherry picked from commit d2c072b) Signed-off-by: Cheng Pan <[email protected]>
|
Thanks, merged to master/1.8 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are the changes needed?
It is basically copied from
kyuubi-extension-spark-3-4.How was this patch tested?
Compiled successfully:
Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before make a pull request
Was this patch authored or co-authored using generative AI tooling?
No