Branch 1.8 fix 5991 #5992
Closed
Branch 1.8 fix 5991 #5992
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… plugins ### _Why are the changes needed?_ Adding `-DskipTests` and `-am` into maven commands ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No. Closes apache#5247 from pan3793/minor. Closes apache#5247 f373fbd [Cheng Pan] Correct building command for Spark authz/lineage plugins Authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]> (cherry picked from commit 522ec94) Signed-off-by: Cheng Pan <[email protected]>
### _Why are the changes needed?_ https://github.com/apache/iceberg/releases/tag/apache-iceberg-1.3.1 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes apache#5250 from yabola/auto_ci. Closes apache#5250 85f2625 [chenliang.lu] [minor] update iceberg version Authored-by: chenliang.lu <[email protected]> Signed-off-by: Cheng Pan <[email protected]> (cherry picked from commit ea44fb8) Signed-off-by: Cheng Pan <[email protected]>
….4.52.v20230823 Fixes apache#5234 Closes apache#5239 from sidhant-nagpal/fix#5234/jetty-upgraded-from-9.4.51.v20230217-to-9.4.52.v20230823. Closes apache#5234 cbd4b49 [Sidhant Nagpal] Merge branch 'master' of https://github.com/sidhant-nagpal/kyuubi into fix#5234/jetty-upgraded-from-9.4.51.v20230217-to-9.4.52.v20230823 608a510 [Sidhant Nagpal] Merge branch 'master' of https://github.com/sidhant-nagpal/kyuubi into fix#5234/jetty-upgraded-from-9.4.51.v20230217-to-9.4.52.v20230823 d2ac35e [Sidhant Nagpal] Revert "apache#5234 Upgrade Jetty from 9.4.51.v20230217 to 9.4.52.v20230823" f15af28 [Sidhant Nagpal] apache#5234 Upgrade Jetty from 9.4.51.v20230217 to 9.4.52.v20230823 2498302 [Sidhant Nagpal] apache#5234 Upgrade Jetty from 9.4.51.v20230217 to 9.4.52.v20230823 Authored-by: Sidhant Nagpal <[email protected]> Signed-off-by: Cheng Pan <[email protected]> (cherry picked from commit 5abc262) Signed-off-by: Cheng Pan <[email protected]>
…Group even though it's in the completed state ### _Why are the changes needed?_ Implement this issue: apache#5232 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes apache#5240 from XorSum/always_cancel_job_group. Closes apache#5232 7da16aa [bkhan] In SparkOperation#cleanup always calls cancelJobGroup even though it's in the completed state Authored-by: bkhan <[email protected]> Signed-off-by: Cheng Pan <[email protected]> (cherry picked from commit 708a0be) Signed-off-by: Cheng Pan <[email protected]>
…batch session list
### _Why are the changes needed?_
```
override def afterEach(): Unit = {
val sessionManager = fe.be.sessionManager.asInstanceOf[KyuubiSessionManager]
sessionManager.allSessions().foreach { session =>
sessionManager.closeSession(session.handle)
}
// there is a chance that `KyuubiBatchService` do batch job submission during this phase
// which causes the above cleanup can not catch all sessions.
sessionManager.getBatchesFromMetadataStore(MetadataFilter(), 0, Int.MaxValue).foreach { batch =>
sessionManager.applicationManager.killApplication(ApplicationManagerInfo(None), batch.getId)
sessionManager.cleanupMetadata(batch.getId)
}
}
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes apache#5246 from pan3793/5138.
Closes apache#5138
a8d6837 [Cheng Pan] try catch
0c418b4 [Cheng Pan] [KYUUBI apache#5138][TEST] Fix flaky test BatchesResourceSuite - get batch session list
Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
(cherry picked from commit 466d35d)
Signed-off-by: Cheng Pan <[email protected]>
…mmand ### _Why are the changes needed?_ Currently, Flink engine doesn't use delegation tokens and these tokens need to be filtered out from the Flink engine launch command, or the command may be corrupted because the credentials could contain new lines. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No. Closes apache#5238 from link3280/filter_engine_credential. Closes apache#5238 5e2403a [Paul Lin] Optimize code style 41df6e2 [Paul Lin] Fix test error 5241894 [Paul Lin] Fix credentials may break Flink engine launch command Authored-by: Paul Lin <[email protected]> Signed-off-by: Cheng Pan <[email protected]> (cherry picked from commit 32c5033) Signed-off-by: Cheng Pan <[email protected]>
### _Why are the changes needed?_ close apache#5253 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ NO Closes apache#5254 from lsm1/branch-kyuubi-5253. Closes apache#5253 b5c4076 [senmiaoliu] bump testcontainers 0.41.0 Authored-by: senmiaoliu <[email protected]> Signed-off-by: Cheng Pan <[email protected]> (cherry picked from commit 6158599) Signed-off-by: Cheng Pan <[email protected]>
### _Why are the changes needed?_ https://github.com/search?q=repo%3Aapache%2Fkyuubi%20incubator-kyuubi&type=code ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes apache#5252 from cxzl25/remove_incubator_link. Closes apache#5252 6485e0b [sychen] remove incubator link Authored-by: sychen <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
…2 to 0.41.0" This reverts commit d5cc678.
…ecovery ### _Why are the changes needed?_ The `recoveryMetadata` is not accurate after batch impl is introduced. This PR proposes to rename `recoveryMetadata` to `metadata` and introduce a dedicated flay `fromRecovery` to distinguish metadata between them. This PR also partially reverts apache#4798, by removing unnecessary constructor parameters `shouldRunAsync` and `batchConf` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No. Closes apache#5243 from pan3793/meta-recov. Closes apache#5243 0718fbe [Cheng Pan] nit b835846 [Cheng Pan] simplify a2d6519 [Cheng Pan] fix test 2dad868 [Cheng Pan] refactor f83d2a6 [Cheng Pan] Distinguish batch impl v2 metadata from recovery Authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]> (cherry picked from commit 6a23f88) Signed-off-by: Cheng Pan <[email protected]>
### _Why are the changes needed?_ close apache#5253 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ NO Closes apache#5254 from lsm1/branch-kyuubi-5253. Closes apache#5253 b5c4076 [senmiaoliu] bump testcontainers 0.41.0 Authored-by: senmiaoliu <[email protected]> Signed-off-by: Cheng Pan <[email protected]> (cherry picked from commit 6158599) Signed-off-by: Cheng Pan <[email protected]>
…ark 3.5 changes ### _Why are the changes needed?_ to fix ``` - columns lineage extract - CreateTableAsSelect *** FAILED *** java.util.NoSuchElementException: None.get at scala.None$.get(Option.scala:529) at scala.None$.get(Option.scala:527) at org.apache.kyuubi.plugin.lineage.helper.SparkSQLLineageParserHelperSuite.extractLineage(SparkSQLLineageParserHelperSuite.scala:1452) at org.apache.kyuubi.plugin.lineage.helper.SparkSQLLineageParserHelperSuite.$anonfun$new$24(SparkSQLLineageParserHelperSuite.scala:354) at org.apache.kyuubi.plugin.lineage.helper.SparkSQLLineageParserHelperSuite.$anonfun$new$24$adapted(SparkSQLLineageParserHelperSuite.scala:353) at org.apache.spark.sql.SparkListenerExtensionTest.withTable(SparkListenerExtensionTest.scala:48) at org.apache.spark.sql.SparkListenerExtensionTest.withTable$(SparkListenerExtensionTest.scala:46) at org.apache.kyuubi.plugin.lineage.helper.SparkSQLLineageParserHelperSuite.withTable(SparkSQLLineageParserHelperSuite.scala:34) at org.apache.kyuubi.plugin.lineage.helper.SparkSQLLineageParserHelperSuite.$anonfun$new$23(SparkSQLLineageParserHelperSuite.scala:353) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ... ``` the method `CreateTableAsSelect#left` was removed since Spark 3.5, for more details, see apache/spark#40734 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes apache#5257 from cfmcgrady/fix-SparkSQLLineageParserHelperSuite. Closes apache#5257 aa5e730 [Fu Chen] fix Authored-by: Fu Chen <[email protected]> Signed-off-by: Cheng Pan <[email protected]> (cherry picked from commit 83b2f83) Signed-off-by: Cheng Pan <[email protected]>
…deps ### _Why are the changes needed?_ Recover CI. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes apache#5258 from pan3793/testcontainers. Closes apache#5253 c1c8241 [Cheng Pan] fix ce4f9ed [Cheng Pan] [KYUUBI apache#5253][FOLLOWUP] Supply testcontainers-scala-scalatest deps for ha module Authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]> (cherry picked from commit 1a925d8) Signed-off-by: Cheng Pan <[email protected]>
### _Why are the changes needed?_ Add Flink KDF doc to close the issue apache#5118 <img width="1094" alt="image" src="https://github.com/apache/kyuubi/assets/3898450/23cd886e-c94a-406b-81a2-eb2caf4c1775"> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes apache#5233 from zhaohehuhu/Improvement-0901. Closes apache#5118 fcd45b3 [hezhao2] code reformat 46bb970 [hezhao2] Add Flink KDF doc Authored-by: hezhao2 <[email protected]> Signed-off-by: Shaoyun Chen <[email protected]> (cherry picked from commit 5ec2c2e) Signed-off-by: Shaoyun Chen <[email protected]>
### _Why are the changes needed?_ Fix: apache#5216 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes apache#5217 from zhuyaogai/issue-5216. Closes apache#5216 b8d2e17 [Fantasy-Jay] Limit counter resource leak in SessionLimiter. cda3702 [Fantasy-Jay] Limit counter resource leak in SessionLimiter. 36272d1 [Fantasy-Jay] fix test bug. 1e282d2 [Fantasy-Jay] Limit counter resource leak in SessionLimiter. 7fc389f [Fantasy-Jay] Limit counter resource leak in SessionLimiter. Authored-by: Fantasy-Jay <[email protected]> Signed-off-by: Cheng Pan <[email protected]> (cherry picked from commit 2371cc6) Signed-off-by: Cheng Pan <[email protected]>
…rization in single call mode ### _Why are the changes needed?_ - improvements for testing by extracting method for enabling authorization in single call mode ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No. Closes apache#5268 from bowenliang123/authz-withsingle. Closes apache#5268 fe6b9d5 [Bowen Liang] extract withSingleCallEnabled method Authored-by: Bowen Liang <[email protected]> Signed-off-by: Bowen Liang <[email protected]> (cherry picked from commit 1a69772) Signed-off-by: Bowen Liang <[email protected]>
### _Why are the changes needed?_ As title, add swagger ui into new kyuubi web ui. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes apache#5263 from zwangsheng/KYUUBI#5259. Closes apache#5259 34c8779 [zwangsheng] keep static swagger ui in kyuubi server 2efbae0 [zwangsheng] fix license 21c8cae [zwangsheng] remove dup dependence and add swagger ui into notive c1bedcb [Cheng Pan] Update kyuubi-server/web-ui/src/views/swagger/index.vue 1820420 [Fu Chen] Update kyuubi-server/web-ui/src/views/swagger/index.vue 05c3b87 [zwangsheng] fix style fc05bb8 [zwangsheng] Add License header e562520 [zwangsheng] Try 9c96800 [zwangsheng] Init swagger page Lead-authored-by: zwangsheng <[email protected]> Co-authored-by: Cheng Pan <[email protected]> Co-authored-by: Fu Chen <[email protected]> Signed-off-by: Cheng Pan <[email protected]> (cherry picked from commit 60b7e8f) Signed-off-by: Cheng Pan <[email protected]>
### _Why are the changes needed?_ <img width="1080" alt="image" src="https://github.com/apache/kyuubi/assets/3898450/cc118574-c752-449a-9b20-04d2cc59b118"> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes apache#5227 from hanna-liashchuk/patch-2. Closes apache#5227 baf391b [Hanna Liashchuk] fix DOC style e46dfdd [Cheng Pan] Update docs/deployment/engine_on_kubernetes.md 0746a35 [hanna-liashchuk] upd 233bba0 [hanna-liashchuk] Added deploy mode info Lead-authored-by: Hanna Liashchuk <[email protected]> Co-authored-by: hanna-liashchuk <[email protected]> Co-authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]> (cherry picked from commit 55feb2b) Signed-off-by: Cheng Pan <[email protected]>
### _Why are the changes needed?_ As title ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No. Closes apache#5273 from pan3793/doc-release. Closes apache#5273 c4f96c6 [Cheng Pan] 1 2266af3 [Cheng Pan] nit 1d52c1a [Cheng Pan] Bump version for web-ui and update release docs Authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]> (cherry picked from commit e27ed44) Signed-off-by: Cheng Pan <[email protected]>
… authz ### _Why are the changes needed?_ To close apache#5269 . 1. AnalyzesTablesCommand in table_command_spec is duplicated and won't be processed 2. The extractor desc of AnalyzesTablesCommand in table_command_spec is wrong ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ Closes apache#5270 from AngersZhuuuu/KYUUBI-5269. Closes apache#5269 75efc1a [Angerszhuuuu] Update TableCommands.scala 75a6fa2 [Angerszhuuuu] Update table_command_spec.json Authored-by: Angerszhuuuu <[email protected]> Signed-off-by: Kent Yao <[email protected]> (cherry picked from commit f967f2f) Signed-off-by: Kent Yao <[email protected]>
…on for golden files ### _Why are the changes needed?_ - Extract common assertion method for verifying file contents - Ensure integrity of the file by comparing the line count - Correct the script name for Spark engine KDF doc generation from `gen_kdf.sh` to `gen_spark_kdf_docs.sh` - Add `gen_hive_kdf_docs.sh` script for Hive engine KDF doc generation - Fix incorrect hints for Ranger spec file generation - shows the line number of the incorrect file content - Streamingly read file content by line with buffered support - Regeneration hints: <img width="656" alt="image" src="https://github.com/apache/kyuubi/assets/1935105/d1a7cb70-8b63-4fe9-ae27-80dadbe84799"> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No. Closes apache#5275 from bowenliang123/doc-regen-hint. Closes apache#5275 9af97ab [Bowen Liang] implicit source position 07020c7 [liangbowen] assertFileContent Lead-authored-by: liangbowen <[email protected]> Co-authored-by: Bowen Liang <[email protected]> Signed-off-by: Bowen Liang <[email protected]> (cherry picked from commit d15322d) Signed-off-by: Bowen Liang <[email protected]>
…rite privilege ### _Why are the changes needed?_ Since AnalyzeTableCommand also update table's metadata, since alter command also need table write privilege, AnalyzeTableCommand need too ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ Closes apache#5272 from AngersZhuuuu/KYUUBI-5271. Closes apache#5271 ad5c704 [Angerszhuuuu] Merge branch 'KYUUBI-5271' of https://github.com/AngersZhuuuu/incubator-kyuubi into KYUUBI-5271 a5932b7 [Angerszhuuuu] Update TableCommands.scala 97ee329 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5271 cdd8100 [Angerszhuuuu] Update PrivilegesBuilderSuite.scala 5f11056 [Angerszhuuuu] update 92c3045 [Angerszhuuuu] Revert "Update TableCommands.scala" 504ff2a [Angerszhuuuu] Update TableCommands.scala 6c42336 [Angerszhuuuu] [KYUUBI apache#5271][Bug] AnalyzeTableCommand should also add table write privilege Authored-by: Angerszhuuuu <[email protected]> Signed-off-by: Kent Yao <[email protected]> (cherry picked from commit b21ae88) Signed-off-by: Kent Yao <[email protected]>
…state ### _Why are the changes needed?_ This PR aims to fix the `SparkSubmit` concurrency limit implemented in apache#5220. "submitted" judgment in apache#5220 only considered `OperationState`, actually, `ApplicationState` should be counted too. For instance, if a batch is pending in `ACCEPTED` state, the `SparkSubmit` process won't exit until changed to `RUNNING` or `FAILED` state, in such case, the `OperationState` is `RUNNING` and `ApplicationState` is `PENDING`, it should not be treated as "submitted". Additionally, this PR treats metastore as the single of truth for batch instead of `SessionManager` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No. Closes apache#5279 from pan3793/5220-followup. Closes apache#5220 903abc6 [Cheng Pan] Fix 0af6738 [Cheng Pan] [KYUUBI apache#5220][FOLLOWUP] Batch submmited should not contain application pending state Authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]> (cherry picked from commit dd2cd51) Signed-off-by: Cheng Pan <[email protected]>
…leak ### _Why are the changes needed?_ ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ Closes apache#5211 from ASiegeLion/master. Closes apache#5210 6366e96 [peiyue liu] Merge branch 'apache:master' into master 34dcc57 [liupeiyue] [KYUUBI apache#5210]Cancel operation will cause the log file node to leak Lead-authored-by: peiyue liu <[email protected]> Co-authored-by: liupeiyue <[email protected]> Signed-off-by: ulyssesyou <[email protected]> (cherry picked from commit 721c0a4) Signed-off-by: ulyssesyou <[email protected]>
### _Why are the changes needed?_ As title We should ref to FIFO design, pick oldest task to run. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes apache#5290 from zwangsheng/KYUUBI/metadata_pick_order. Closes apache#5290 2d3b096 [zwangsheng] [REST-V2] Should use ASC order when kyuubi server submitter pick task from metadata Authored-by: zwangsheng <[email protected]> Signed-off-by: Cheng Pan <[email protected]> (cherry picked from commit f6e3225) Signed-off-by: Cheng Pan <[email protected]>
### _Why are the changes needed?_ Previously, the RESTful audit log did not contain HTTP requests that threw non-AuthenticationException during the process. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No. Closes apache#5289 from pan3793/auth-log. Closes apache#5289 9e29279 [Cheng Pan] RESTful API should always print audit log Authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]> (cherry picked from commit 137dcf1) Signed-off-by: Cheng Pan <[email protected]>
### _Why are the changes needed?_ upgrade snakeyaml from 1.33 to 2.2 reducing direct CVE vulnerabilities, see (https://bitbucket.org/snakeyaml/snakeyaml/wiki/Changes) [CVE-2022-1471](https://nvd.nist.gov/vuln/detail/CVE-2022-1471) SnakeYaml's Constructor() class does not restrict types which can be instantiated during deserialization. Deserializing yaml content provided by an attacker can lead to remote code execution. We recommend using SnakeYaml's SafeConsturctor when parsing untrusted content to restrict deserialization. We recommend upgrading to version 2.0 and beyond. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes apache#5293 from dev-lpq/snakeyaml_critical. Closes apache#5293 5b2412d [pengqli] upgrade snakeyaml from 1.33 to 2.2 Authored-by: pengqli <[email protected]> Signed-off-by: Cheng Pan <[email protected]> (cherry picked from commit 9b93e87) Signed-off-by: Cheng Pan <[email protected]>
### _Why are the changes needed?_ Apache Spark 3.5.0 released in few days ago, this PR aims to add cross-version verification to ensure that Kyuubi engine built against Spark 3.4.1 works on Spark 3.5.0 runtime. https://spark.apache.org/releases/spark-release-3-5-0.html For users who want to try Kyuubi with Spark 3.5.0, please compile master/branch-1.8 via ``` ./build/dist --tgz --web-ui --spark-provided --flink-provided --hive-provided ``` And then follow the [Quick Start](https://kyuubi.readthedocs.io/en/master/quick_start/quick_start.html) to setup Kyuubi and Spark properly. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate ```log Connected to: Spark SQL (version 3.5.0) Driver: Kyuubi Project Hive JDBC Client (version 1.8.0-SNAPSHOT) Beeline version 1.8.0-SNAPSHOT by Apache Kyuubi 0: jdbc:hive2://0.0.0.0:10009/default> select version(); +-------------------------------------------------+ | version() | +-------------------------------------------------+ | 3.5.0 ce5ddad990373636e94071e7cef2f31021add07b | +-------------------------------------------------+ 1 row selected (0.39 seconds) 0: jdbc:hive2://0.0.0.0:10009/default> select kyuubi_version(); +-------------------+ | kyuubi_version() | +-------------------+ | 1.8.0-SNAPSHOT | +-------------------+ 1 row selected (0.16 seconds) ``` - [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No. Closes apache#5291 from pan3793/cross-3.5. Closes apache#5291 24107a4 [Cheng Pan] Update .github/workflows/master.yml 1554e71 [Cheng Pan] Verify Spark engine on Spark 3.5 Authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]> (cherry picked from commit 7ba458a) Signed-off-by: Cheng Pan <[email protected]>
…ver uri from metadata.kyuubiInstance ### _Why are the changes needed?_ Due to apache#5078, we marked kyuubi_instance in metadata can be null. We should append kyuubi_instance after we check it's non-null. Otherwise we may face the following error when we use v2 submit job ``` 2023-09-14 19:05:43 [INFO] [main] org.apache.kyuubi.client.RetryableRestClient#74 - Current connect server uri http://null/api/v1 2023-09-14 19:05:43 [ERROR] [main] org.apache.kyuubi.client.RestClient#189 - Error: java.net.UnknownHostException: null: Name or service not known ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes apache#5297 from zwangsheng/client/resetclient_get_null. Closes apache#5297 2b4b560 [zwangsheng] [CLIENT] Get null from metadata.kyuubiInstance Authored-by: zwangsheng <[email protected]> Signed-off-by: Cheng Pan <[email protected]> (cherry picked from commit 60ebe7d) Signed-off-by: Cheng Pan <[email protected]>
…open session ### _Why are the changes needed?_ See more in close apache#5300 Remove update metadata kyuubi_instance, when open session under batch v2 implementation. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes apache#5301 from zwangsheng/server/v2_should_not_update_connect_url. Closes apache#5300 0a67bcd [zwangsheng] [V2][REST] Not update kyuubi instace when open session Authored-by: zwangsheng <[email protected]> Signed-off-by: Cheng Pan <[email protected]> (cherry picked from commit 4136835) Signed-off-by: Cheng Pan <[email protected]>
# 🔍 Description ## Issue References 🔗 This pull request fixes apache#5969 ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes apache#5971 from lsm1/branch-kyuubi-trino-schema. Closes apache#5969 dfc987e [senmiaoliu] trino engine add default result schema Authored-by: senmiaoliu <[email protected]> Signed-off-by: Cheng Pan <[email protected]> (cherry picked from commit fcad8b5) Signed-off-by: Cheng Pan <[email protected]>
…omTicketCache
# 🔍 Description
The use case:
- 1. the user code running in flink cluster and wrapped by transparent `ugi.doAs`
- 2. the current UserGroupInfomation is the proxy user not login/real user
- 3. user specify url with kyuubiClientTicketCache but does not work
- 4. the proxy UserGroupInfomation is used and then throw GSS issue.
```
Caused by: org.apache.kyuubi.shade.org.apache.thrift.transport.TTransportException: GSS initiate failed
at org.apache.kyuubi.shade.org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
at org.apache.kyuubi.shade.org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316)
at org.apache.kyuubi.shade.org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at org.apache.kyuubi.jdbc.hive.auth.TSubjectTransport.lambda$open$0(TSubjectTransport.java:47)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.kyuubi.jdbc.hive.auth.TSubjectTransport.open(TSubjectTransport.java:42)
at org.apache.kyuubi.jdbc.hive.KyuubiConnection.openTransport(KyuubiConnection.java:458)
at org.apache.kyuubi.jdbc.hive.KyuubiConnection.<init>(KyuubiConnection.java:207)
... 22 more
```
The root cause is that, for this case, the result of `isHadoopUserGroupInformationDoAs` is true.
So, `isFromSubjectAuthMode` is true.
In this pr, I want to specify the kerberosAuthType to fromTicketCache, and do not check `isHadoopUserGroupInformationDoAs`.
After this pr, customer can specify `kerberosAuthType=fromTicketCache` to leverage ticket cache prefer than `ugi.doAs`.
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes apache#5961 from turboFei/warn_ticket_cache.
Closes apache#5961
6e06500 [Fei Wang] exception
c01a99e [Fei Wang] refine
098a37b [Fei Wang] do not infer fromSubject if fromTicketCache && ugi
042fa22 [Fei Wang] warn
Authored-by: Fei Wang <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
(cherry picked from commit 54086b0)
Signed-off-by: Cheng Pan <[email protected]>
…ntSession
# 🔍 Description
## Issue References 🔗
This pull request fixes argument position on constructing Trino `ClientSession`.
## Describe Your Solution 🔧
Invert `properties` and `Collections.emptyMap()` to match the `ClientSession` constructor signature.
```
public ClientSession(
URI server,
String principal,
Optional<String> user,
String source,
Optional<String> traceToken,
Set<String> clientTags,
String clientInfo,
String catalog,
String schema,
String path,
ZoneId timeZone,
Locale locale,
Map<String, String> resourceEstimates,
Map<String, String> properties,
Map<String, String> preparedStatements,
Map<String, ClientSelectedRole> roles,
Map<String, String> extraCredentials,
String transactionId,
Duration clientRequestTimeout,
boolean compressionDisabled)
```
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes apache#5977 from pan3793/trino-props-arg.
Closes apache#5977
1e0f71a [Cheng Pan] Fix argument position on constructing Trino ClientSession
Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
(cherry picked from commit dbf70d4)
Signed-off-by: Cheng Pan <[email protected]>
… exists # 🔍 Description Followup apache#5961 ``` scala> Files.exists(Paths.get(null)) <console>:14: error: ambiguous reference to overloaded definition, both method get in class Paths of type (x$1: java.net.URI)java.nio.file.Path and method get in class Paths of type (x$1: String, x$2: String*)java.nio.file.Path match argument types (Null) and expected result type java.nio.file.Path Files.exists(Paths.get(null)) ^ scala> Files.exists(Paths.get("")) res0: Boolean = true scala> ``` ## Issue References 🔗 This pull request fixes # ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes :bookmark: - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request :coffin: #### Behavior With This Pull Request :tada: #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes apache#5980 from turboFei/fix_npe. Closes apache#5961 99d4a16 [Fei Wang] fix npe Authored-by: Fei Wang <[email protected]> Signed-off-by: Cheng Pan <[email protected]> (cherry picked from commit cbe177b) Signed-off-by: Cheng Pan <[email protected]>
# 🔍 Description ## Issue References 🔗 This pull request canonicalizes Trino IT in GitHub Action workflow. `mvn -am` means "also make", so that `mvn integration-tests/kyuubi-trino-it -am` would trigger compiling of `kyuubi-server`, `externals/kyuubi-spark-sql-engine`, `externals/kyuubi-download` modules automatically. ## Describe Your Solution 🔧 CI part change in apache@6688b3d is unnecessary. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Pass GA. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes apache#5978 from pan3793/trino-ci. Closes apache#5978 1b8ef94 [Cheng Pan] set KYUUBI_HOME in trino-it 18e72d3 [Cheng Pan] Canonicalize Trino IT in GitHub Action workflow Authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]> (cherry picked from commit 3915fe8) Signed-off-by: Cheng Pan <[email protected]>
github-actions
bot
added
module:kubernetes
kind:documentation
|
@SwordyZhao please make the change on the latest master branch, for bug fix, the committer will help to evaluate if the backport is applicable |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
🔍 Description
Issue References 🔗
This pull request fixes #5991
Describe Your Solution 🔧
This error occurs because when the java class ApplicationProperties reads the configuration, configuration items with multiple values are returned as a java List. When kyuubi handles the configuration, it does not handle the java List specifically, so it calls the toString method, and the toString of java List returns a string with brackets, resulting in a configuration read error
Types of changes 🔖
Test Plan 🧪
Behavior Without This Pull Request ⚰️
Behavior With This Pull Request 🎉
Related Unit Tests
Checklist 📝
Be nice. Be informative.