[pull] master from apache:master #36
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![pull[bot]](https://avatars.githubusercontent.com/in/12910?s=60&v=4){kind=small}
# 🔍 Description ## Issue References 🔗 This pull request fixes # ## Describe Your Solution 🔧 Return `sessionName` and `totalOperations` within `SessionData`. Return `appStartTime` with list batches. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6359 from turboFei/total_op. Closes #6359 5fb3fb8 [Wang, Fei] fix compile fc39705 [Wang, Fei] comments 0e7ff00 [Wang, Fei] return app time bb4b649 [Wang, Fei] dto Authored-by: Wang, Fei <[email protected]> Signed-off-by: Wang, Fei <[email protected]>
…dsException # 🔍 Description ## Issue References 🔗 This pull request fixes #6370 ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6372 from turboFei/kyuubi_6370. Closes #6370 d0c6a92 [Wang, Fei] fix Authored-by: Wang, Fei <[email protected]> Signed-off-by: Wang, Fei <[email protected]>
…eToFile # 🔍 Description ## Issue References 🔗 This pull request fixes # I found that, with saveToFile enabled with the default min size threshold, even I run a simple `set` command, It also save the result to file. <img width="1718" alt="image" src="https://github.com/apache/kyuubi/assets/6757692/5bcc0da1-201a-453a-8568-d1bfadd7adef"> I think we need to skip this kind of queries. ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6377 from turboFei/check_is_DQL. Closes #6377 da9c2a9 [Wang, Fei] ut 04e20db [Wang, Fei] conf 8f20ed8 [Wang, Fei] refine the check f558dcc [Wang, Fei] ut c813403 [Wang, Fei] DQL Authored-by: Wang, Fei <[email protected]> Signed-off-by: Wang, Fei <[email protected]>
# 🔍 Description ## Issue References 🔗 This pull request fixes #6373 ## Describe Your Solution 🔧 `ThriftUtils#EMPTY_ROW_SET` is not suitable for OperationLog , so I add a new `LOG_EMPTY_ROW_SET` ## Types of changes 🔖 - [X] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6378 from wForget/KYUUBI-6373. Closes #6373 f25aa59 [wforget] revert ef87b3f [wforget] revert and fix npe 7219d35 [wforget] add test 06135fe [wforget] fix test c0bd290 [wforget] [KYUUBI #6373] Add empty RowSet for OpertationLog Authored-by: wforget <[email protected]> Signed-off-by: Wang, Fei <[email protected]>
…mediately after marking the engine not alive Support to interrupt the thrift request immediately after marking the engine not alive # 🔍 Description ## Issue References 🔗 This pull request fixes #6172 ## Describe Your Solution 🔧 https://github.com/apache/kyuubi/blob/12c5568c9b020b1212c9514f046c67fcb267467e/kyuubi-server/src/main/scala/org/apache/kyuubi/client/KyuubiSyncThriftClient.scala#L103-L110 When probe fails and exceeds engineAliveTimeout, not interrupt the thrift request immediately, only marked `remoteEngineBroken` and wait next `engineAliveProbeInterval` to interrupt. Unit test `KyuubiOperationPerConnectionSuite` assert timeout 3s. https://github.com/apache/kyuubi/blob/12c5568c9b020b1212c9514f046c67fcb267467e/kyuubi-server/src/test/scala/org/apache/kyuubi/operation/KyuubiOperationPerConnectionSuite.scala#L344-L346 Exception log ``` 03:25:15.125 engine-alive-probe-TSessionHandle(sessionId:THandleIdentifier(guid:BD 23 DE B6 16 56 4E 2B 97 3C 09 74 89 F5 C9 26, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38)): Thread-11886 WARN KyuubiSyncThriftClient: The engine[local-1714879506640] alive probe fails org.apache.kyuubi.shaded.thrift.transport.TTransportException: Socket is closed by peer. ... 03:25:16.126 engine-alive-probe-TSessionHandle(sessionId:THandleIdentifier(guid:BD 23 DE B6 16 56 4E 2B 97 3C 09 74 89 F5 C9 26, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38)): Thread-11886 WARN KyuubiSyncThriftClient: The engine[local-1714879506640] alive probe fails org.apache.kyuubi.shaded.thrift.transport.TTransportException: java.net.SocketException: Broken pipe (Write failed) ... 03:25:16.126 engine-alive-probe-TSessionHandle(sessionId:THandleIdentifier(guid:BD 23 DE B6 16 56 4E 2B 97 3C 09 74 89 F5 C9 26, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38)): Thread-11886 ERROR KyuubiSyncThriftClient: Mark the engine[local-1714879506640] not alive with no recent alive probe success: 2001 ms exceeds timeout 1000 ms ``` Success log ``` 16:57:46.859 engine-alive-probe-TSessionHandle(sessionId:THandleIdentifier(guid:B4 9D 71 83 6D 15 4D 8D BE DA 65 75 27 5D 4E D8, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38)): Thread-11609 WARN KyuubiSyncThriftClient: The engine[local-1715101059872] alive probe fails ... 16:57:46.860 engine-alive-probe-TSessionHandle(sessionId:THandleIdentifier(guid:B4 9D 71 83 6D 15 4D 8D BE DA 65 75 27 5D 4E D8, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38)): Thread-11609 ERROR KyuubiSyncThriftClient: Mark the engine[local-1715101059872] not alive with no recent alive probe success: 1001 ms exceeds timeout 1000 ms 16:57:47.860 engine-alive-probe-TSessionHandle(sessionId:THandleIdentifier(guid:B4 9D 71 83 6D 15 4D 8D BE DA 65 75 27 5D 4E D8, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38)): Thread-11609 WARN KyuubiSyncThriftClient: Removing Clients for TSessionHandle(sessionId:THandleIdentifier(guid:9D AA D5 C2 9B E4 43 D7 BE 81 D0 99 EA 5B 9E 37, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38)) ``` ## Types of changes :bookmark: - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request :coffin: #### Behavior With This Pull Request :tada: #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6375 from beryllw/kyuubi_6172. Closes #6172 991798b [wangjunbo] [KYUUBI #6172][TASK][EASY] Support to interrupt the thrift request immediately after marking the engine not alive Authored-by: wangjunbo <[email protected]> Signed-off-by: Wang, Fei <[email protected]>
…ing checker # 🔍 Description ## Issue References 🔗 This pull request fixes # follow up of #4847 Address comments: #4847 (comment) ## Describe Your Solution 🔧 In this pr, when checking the engine terminating, it will ignore the alive probe sessions. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6355 from turboFei/engine_idle. Closes #4847 a8e26e7 [Wang, Fei] comments 418d0b4 [Wang, Fei] val Authored-by: Wang, Fei <[email protected]> Signed-off-by: Wang, Fei <[email protected]>
…Event/SessionData # 🔍 Description ## Issue References 🔗 In this pr, engineName and engineUrl will return within KyuubiSessionEvent/SessionData, these information are helpful to get the session info straight forward. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6379 from turboFei/app_name_session_event. Closes #6379 f9c4b0d [Wang, Fei] refine 15b4cfc [Wang, Fei] engine info 4662878 [Wang, Fei] id name url 9e1d72b [Wang, Fei] id name url Authored-by: Wang, Fei <[email protected]> Signed-off-by: Wang, Fei <[email protected]>
# 🔍 Description Hive 2.3.10 is available now, and it will be the latest version of the 2.3 serial. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Pass GA. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6328 from pan3793/hive-2310. Closes #6328 82a9fb7 [Cheng Pan] Update .github/workflows/master.yml 10af5e1 [Cheng Pan] hive-2.3.10-rc-1 2dbb2b5 [Cheng Pan] test Hive 2.3.10 RC0 Authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
# 🔍 Description ## Issue References 🔗 This pull request fixes #6348 ## Describe Your Solution 🔧 In the project POM file, I have updated the Apache Iceberg version from 1.5.0 to 1.5.2 ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6349 from mrpengbei/KYUUBI6348. Closes #6348 9303898 [Peng Bei] Update pom.xml 119bb78 [pengbei] [KYUUBI #6348] Upgrade iceberg from 1.5.0 to 1.5.1 #6348 Lead-authored-by: pengbei <[email protected]> Co-authored-by: Peng Bei <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
…Session # 🔍 Description ## Issue References 🔗 This pull request aims to improve closeEngine method names and logs due to closeSession. when the share level is `CONNECTION`, closeSession triggers the closeEngine, there are some minor issues with logging and method name. ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6376 from yikf/rename-close-session. Closes #6376 b03661b [yikaifei] rename closeSession Authored-by: yikaifei <[email protected]> Signed-off-by: yikaifei <[email protected]>
# 🔍 Description ## Issue References 🔗 [Delta 3.2.0](https://github.com/delta-io/delta/releases/tag/v3.2.0) is available, which is built on top of Spark 3.5. ## Describe Your Solution 🔧 Bump Delta from 3.1.0 to 3.2.0 for Spark 3.5. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Pass GA. #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6382 from zml1206/delta-3.2. Closes #6382 6de96c6 [zml1206] Bump Delta from 3.1.0 to 3.2.0 for Spark 3.5 Authored-by: zml1206 <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
# 🔍 Description This PR removes two dependencies from the `kyuubi-rest-client` module - `commons-collections` - has CVE Cx78f40514-81ff and is only used in one place, just rewrite to remove the dependency - `javax.servlet-api` - only used for UT, correct the scope from `compile` to `test` ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Pass GHA --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6457 from pan3793/rest-client-dep. Closes #6457 c24af70 [Cheng Pan] dep list 80969d0 [Cheng Pan] nit 950d4b7 [Cheng Pan] fix 1692819 [Cheng Pan] fix import c068ba1 [Cheng Pan] nit d46653d [Cheng Pan] fix ca78317 [Cheng Pan] Strip dependencies from REST client Authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
# 🔍 Description [`jcl-over-slf4j`](https://www.slf4j.org/legacy.html#jcl-over-slf4j) is a drop-in replacement of `commons-logging`, the latter one should not be present in the final classpath, otherwise, there are potential class conflict issues. The current dep check is problematic, this PR also changes it to always perform "install" to fix the false negative report. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Simply delete `commons-logging-1.1.3.jar` from `apache-kyuubi-1.9.1-bin.tgz` and everything goes well. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6458 from pan3793/commons-logging. Closes #6458 114ec76 [Cheng Pan] fix 79d4121 [Cheng Pan] fix 6633e83 [Cheng Pan] fix 21127ed [Cheng Pan] always perform install on dep check 98b13df [Cheng Pan] Remove commons-logging from binary release Authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
# 🔍 Description ## Issue References 🔗 The concurrency limit for the engine startup process is mainly used to avoid overload on the machine(or container) of the Kyuubi server, the current implementation holds startupProcessSemaphore until the session is established successfully. While for Spark on YARN cluster mode, some YARN queue resource insufficiency may block the subsequent Spark application submissions to other queues, significantly affecting the Kyuubi server's resource utilization. ## Describe Your Solution 🔧 We should immediately release the `startupProcessSemaphore` after the engine startup process exits (i.e., after the `spark-submit` process exits) as the load has already disappeared. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 I tested it on a cluster of 50 kyuubi Servers, and kyuubi server resource utilization increased by 70% --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6463 from ic4y/master-p003. Closes #6463 f7de68c [ic4y] Improve code quality d8b0248 [ic4y] [Improve][EngineRef] Optimize Engine Startup Concurrency Limit Authored-by: ic4y <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
# 🔍 Description ## Issue References 🔗 This pull request fixes # ## Describe Your Solution 🔧 Bump minikube and kubernetes version of integration test to fix failure of integration test in CI. - minikube: v1.29.0->v1.33.1 - kubernetes: v1.26.1->v1.30.0 Docker version 25 loads images into various distributions of Kubernetes has been impossible due to what appears to be mismatching hashes of manifests/images as follows: ``` X Exiting due to GUEST_IMAGE_LOAD: save to dir: caching images: caching image "/home/runner/.minikube/cache/images/amd64/apache/kyuubi_latest": write: unable to calculate manifest: blob sha256:5bbc241b2d6dcc55b5a63c080556ad458ba1395e93af3204d12e72c9a192eb06 not found ``` Minikube and kubernetes version should be bumped to fix failure of integration test which refers to moby/moby#47207. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6466 from cfmcgrady/bump-minikube. Closes #6466 c4bf4b5 [Fu Chen] Bump minikube and kubernetes version of integration test Authored-by: Fu Chen <[email protected]> Signed-off-by: Fu Chen <[email protected]>
# 🔍 Description ## Issue References 🔗 Address comments #4847 (comment) Now, for `checkEngineConnectionAlive`, it use the client to send `TGetInfoType` to engine and cause the user session never idle for timeout. ## Describe Your Solution 🔧 We shall reuse the alive probe client. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests Pass the current UT. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6468 from turboFei/engine_alive_check. Closes #4847 e2368b2 [Wang, Fei] reuse Authored-by: Wang, Fei <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
…ion finished # 🔍 Description ## Issue References 🔗 This pull request fixes #6302 : when the SQL is submitted by Kyuubi Rest Api (the same kyuubi path with BEELINE), the legacy job will not be cancelled, #6302 (comment) . The reason: Beeline session calls `cancelJobGroup` in `SparkOperation#cleanup`. But Rest Api session doesn't call `SparkOperation#cleanup`, so the legacy job submitted by Rest Api will not be canceled. ## Describe Your Solution 🔧 The modification: call `SparkOperation#cleanup` in `ExecuteStatement#executeStatement`'s finally clause. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6473 from XorSum/features/cancel-after-execute-stmt. Closes #6302 16dd508 [xorsum] operation executeStatement cancel group Authored-by: xorsum <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
…ntal collect mode # 🔍 Description ## Issue References 🔗 This pull request fixes #6473 (comment) ## Describe Your Solution 🔧 add a configuration to control whether to skip the cancellation here for incremental collect queries, skipping by default for safety. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6482 from XorSum/features/skip-cancel-incremental. Closes #6302 440311f [xorsum] reformat edbc378 [bkhan] Update externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/ExecuteStatement.scala d6c9936 [xorsum] one line 9f40405 [xorsum] update configuration b152631 [xorsum] skip job group cancellation on incremental collect mode Lead-authored-by: xorsum <[email protected]> Co-authored-by: bkhan <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
Bumps [ws](https://github.com/websockets/ws) from 8.8.1 to 8.17.1. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/websockets/ws/releases">ws's releases</a>.</em></p> <blockquote> <h2>8.17.1</h2> <h1>Bug fixes</h1> <ul> <li>Fixed a DoS vulnerability (<a href="https://redirect.github.com/websockets/ws/issues/2231">#2231</a>).</li> </ul> <p>A request with a number of headers exceeding the[<code>server.maxHeadersCount</code>][] threshold could be used to crash a ws server.</p> <pre lang="js"><code>const http = require('http'); const WebSocket = require('ws'); <p>const wss = new WebSocket.Server({ port: 0 }, function () { const chars = "!#$%&'*+-.0123456789abcdefghijklmnopqrstuvwxyz^_`|~".split(''); const headers = {}; let count = 0;</p> <p>for (let i = 0; i < chars.length; i++) { if (count === 2000) break;</p> <pre><code>for (let j = 0; j &lt; chars.length; j++) { const key = chars[i] + chars[j]; headers[key] = 'x'; if (++count === 2000) break; } </code></pre> <p>}</p> <p>headers.Connection = 'Upgrade'; headers.Upgrade = 'websocket'; headers['Sec-WebSocket-Key'] = 'dGhlIHNhbXBsZSBub25jZQ=='; headers['Sec-WebSocket-Version'] = '13';</p> <p>const request = http.request({ headers: headers, host: '127.0.0.1', port: wss.address().port });</p> <p>request.end(); }); </code></pre></p> <p>The vulnerability was reported by <a href="https://github.com/rrlapointe">Ryan LaPointe</a> in <a href="https://redirect.github.com/websockets/ws/issues/2230">websockets/ws#2230</a>.</p> <p>In vulnerable versions of ws, the issue can be mitigated in the following ways:</p> <ol> <li>Reduce the maximum allowed length of the request headers using the [<code>--max-http-header-size=size</code>][] and/or the [<code>maxHeaderSize</code>][] options so that no more headers than the <code>server.maxHeadersCount</code> limit can be sent.</li> </ol> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/websockets/ws/commit/3c56601092872f7d7566989f0e379271afd0e4a1"><code>3c56601</code></a> [dist] 8.17.1</li> <li><a href="https://github.com/websockets/ws/commit/e55e5106f10fcbaac37cfa89759e4cc0d073a52c"><code>e55e510</code></a> [security] Fix crash when the Upgrade header cannot be read (<a href="https://redirect.github.com/websockets/ws/issues/2231">#2231</a>)</li> <li><a href="https://github.com/websockets/ws/commit/6a00029edd924499f892aed8003cef1fa724cfe5"><code>6a00029</code></a> [test] Increase code coverage</li> <li><a href="https://github.com/websockets/ws/commit/ddfe4a804d79e7788ab136290e609f91cf68423f"><code>ddfe4a8</code></a> [perf] Reduce the amount of <code>crypto.randomFillSync()</code> calls</li> <li><a href="https://github.com/websockets/ws/commit/b73b11828d166e9692a9bffe9c01a7e93bab04a8"><code>b73b118</code></a> [dist] 8.17.0</li> <li><a href="https://github.com/websockets/ws/commit/29694a5905fa703e86667928e6bacac397469471"><code>29694a5</code></a> [test] Use the <code>highWaterMark</code> variable</li> <li><a href="https://github.com/websockets/ws/commit/934c9d6b938b93c045cb13e5f7c19c27a8dd925a"><code>934c9d6</code></a> [ci] Test on node 22</li> <li><a href="https://github.com/websockets/ws/commit/1817bac06e1204bfb578b8b3f4bafd0fa09623d0"><code>1817bac</code></a> [ci] Do not test on node 21</li> <li><a href="https://github.com/websockets/ws/commit/96c9b3deddf56cacb2d756aaa918071e03cdbc42"><code>96c9b3d</code></a> [major] Flip the default value of <code>allowSynchronousEvents</code> (<a href="https://redirect.github.com/websockets/ws/issues/2221">#2221</a>)</li> <li><a href="https://github.com/websockets/ws/commit/e5f32c7e1e6d3d19cd4a1fdec84890e154db30c1"><code>e5f32c7</code></a> [fix] Emit at most one event per event loop iteration (<a href="https://redirect.github.com/websockets/ws/issues/2218">#2218</a>)</li> <li>Additional commits viewable in <a href="https://github.com/websockets/ws/compare/8.8.1...8.17.1">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `dependabot rebase` will rebase this PR - `dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `dependabot merge` will merge this PR after your CI passes on it - `dependabot squash and merge` will squash and merge this PR after your CI passes on it - `dependabot cancel merge` will cancel a previously requested merge and block automerging - `dependabot reopen` will reopen this PR if it is closed - `dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/kyuubi/network/alerts). </details> Closes #6484 from dependabot[bot]/dependabot/npm_and_yarn/kyuubi-server/web-ui/ws-8.17.1. Closes #6484 58a7da8 [dependabot[bot]] ⬆️ Bump ws from 8.8.1 to 8.17.1 in /kyuubi-server/web-ui Authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Cheng Pan <[email protected]>
# 🔍 Description ## Issue References 🔗 This pull request fixes #6476 : spark historyserver -> Show incomplete applications -> kyuubi query engine ui error（java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.Long）. The reason: it's related to FasterXML/jackson-module-scala#62 ## Describe Your Solution 🔧 add JsonDeserialize(contentAs = classOf[java.lang.Long]) annotation ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6479 from felixzh2020/issues/6476. Closes #6476 034bfe5 [felixzh] [KYUUBI #6476] spark historyserver Show incomplete applications kyuubi query engine ui error b7b0db2 [felixzh] [KYUUBI #6476] spark historyserver Show incomplete applications kyuubi query engine ui error a66163a [felixzh] [KYUUBI #6476] spark historyserver Show incomplete applications kyuubi query engine ui error Authored-by: felixzh <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
… oom when fetching big result set # 🔍 Description ## Issue References 🔗 This pull request fixes #6469 ## Describe Your Solution 🔧 Instead of initializing all RecordReaderIterator when create OrcFileIterator，we can lazily initialize the RecordReaderIterator to make sure that there is only one RecordReaderIterator which reads file current fetching by client in driver memory. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) Closes #6470 from Z1Wu/bugfix-fetch-big-resultset-lazily. Closes #6469 8320801 [吴梓溢] update 56284e6 [吴梓溢] update Authored-by: 吴梓溢 <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
…SQL dialect # 🔍 Description ## Issue References 🔗 This pull request fixes #6489 ## Describe Your Solution 🔧 After my investigation, I found the bug and solution. The function get_table_names returns an incorrect value when I used Superset to connect to Kyuubi for Spark SQL. [get_table_names](https://github.com/apache/kyuubi/blob/master/python/pyhive/sqlalchemy_hive.py#L380) The following code is used to connect to hive directly. `return [row[0] for row in connection.execute(text(query))]` Because The following value is returned when the Hive is connected. show tables in default : [('student',), ('student_scores',)] The following code is used to connect to Kyuubi. `return [row[1] for row in connection.execute(text(query))]` Because The following value is returned when the Kyuubi is connected. show tables in default : [('default', 'employees', False), ('default', 'student', False), ('default', 'student_scores', False)] So, for the difference in return value, I modified the code. And I test them in Superset. The code works. Hive <img width="1214" alt="image" src="https://github.com/apache/kyuubi/assets/29974394/9048b21d-053e-4b5d-be35-ba29d3bd6848"> Kyuubi <img width="1085" alt="image" src="https://github.com/apache/kyuubi/assets/29974394/d600dfed-1127-41ea-a0bf-ca662a5487df"> Spark SQL also works properly. <img width="1199" alt="image" src="https://github.com/apache/kyuubi/assets/29974394/7026e39e-6d63-473d-9e43-eeab580719ea"> ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6490 from BruceWong96/branch-kyuubi-6489. Closes #6489 94a52c0 [wenjie.wang01] add else branch. 8ab20be [wenjie.wang01] fix bug for function get_table_names. 136c7b7 [wenjie.wang01] fix bug for function get_table_names. Authored-by: wenjie.wang01 <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
Small notice change to include that pyhive was originally developed at Dropbox. Closes #6434 from bkyryliuk/bkyryliuk-patch-1. Closes #6434 d2ed97b [Bogdan] Add footnote about pyhive origin Authored-by: Bogdan <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
# 🔍 Description This PR rewrites some utility methods in Java, specifically, ``` Utils.isWindows Utils.isMac Utils.findLocalInetAddress ``` and moves them from `kyuubi-common`'s `Utils` to the `kyuubi-util`'s `JavaUtils`, so that they could be used in other modules that do not depend on `kyuubi-common`. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Pass GHA. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6499 from pan3793/javautils. Closes #6499 565936d [Cheng Pan] fix f06a85e [Cheng Pan] Move some untiliy methods in Java Authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
…egationTokenIdentifier # 🔍 Description It was fixed in apache/kyuubi-shaded#44 ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Pass GHA. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6500 from pan3793/rm-hms-dt-identifiler. Closes #6500 f535e3f [Cheng Pan] fix 47b32e3 [Cheng Pan] Remove org.apache.kyuubi.shaded.hive.metastore.security.DelegationTokenIdentifier Authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
…faults.conf # 🔍 Description ## Issue References 🔗 as title This pull request closes #6255 ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6339 from zhaohehuhu/dev-0426. Closes #6339 38b36f2 [Cheng Pan] fix b2e3fcf [Cheng Pan] rewrite Lead-authored-by: hezhao2 <[email protected]> Co-authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
# 🔍 Description ## Issue References 🔗 Default Kyuubi version in the chart is not up to date with the latest release version. ## Describe Your Solution 🔧 Upgrade default version to the latest 1.9.1 to be up to date with the release version. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### The chart deploys Kyuubi 1.9.1 Install the chart: ```sh helm install kyuubi charts/kyuubi ``` Check version: ```sh kubectl exec kyuubi-0 -- cat /opt/kyuubi/RELEASE Kyuubi 1.9.1 (git revision c232a4b) built for Java 1.8.0_412 Scala 2.12 Flink 1.17.2 Spark 3.5.1 Kyuubi Hadoop 3.3.6 Hive 3.1.3 Build flags: ``` --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6493 from dnskr/helm-default-kyuubi-version-1.9.1. Closes #6493 e9b7406 [dnskr] [K8S][HELM] Update default Kyuubi version to 1.9.1 Authored-by: dnskr <[email protected]> Signed-off-by: dnskr <[email protected]>
# 🔍 Description Update the outdated docs to mention new features: 1. Arrow serialization 2. Sava query results into files ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Docs changes, review. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6504 from pan3793/docs-large-result. Closes #6504 77043bd [Cheng Pan] fix e8ba1ba [Cheng Pan] grammar 198d00e [Cheng Pan] grammar a37f631 [Cheng Pan] Improve docs for Solution for Large Query Results Authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
# 🔍 Description Hive 2.3.0 (HIVE-7224) turns it on by default, while Kyuubi BeeLine is a fork from Hive BeeLine 3.1.3, we should update the help message to reflect the change. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Pass GHA. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6507 from pan3793/beeline-incr. Closes #6507 8b1ed65 [Cheng Pan] fix 7a872ee [Cheng Pan] fix 88e751d [Cheng Pan] Correct Kyuubi BeeLine help message for incremental Authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
# 🔍 Description ## Issue References 🔗 This pull request fixes #6506 ## Describe Your Solution 🔧 Set the alias `--conf` to `--hiveconf`. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ ``` # use --hiveconf or --hivevar to set some configurations kyuubi-beeline -u 'jdbc:kyuubi://kyuubi:10009/' \ --hiveconf kyuubi.operation.result.format=arrow \ --hiveconf kyuubi.operation.incremental.collect=true-f \ --hivevar app_name=xxx \ xxx.sql ``` #### Behavior With This Pull Request 🎉 ``` # use --conf to set some configurations kyuubi-beeline -u 'jdbc:kyuubi://kyuubi:10009/' \ --conf kyuubi.operation.result.format=arrow \ --conf kyuubi.operation.incremental.collect=true-f \ --conf app_name=xxx \ xxx.sql ``` #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6511 from BruceWong96/support-beeline-conf. Closes #6506 7a81455 [Bruce Wong] fix code style. 8d1fec2 [Bruce Wong] delete some spaces. d64505d [Bruce Wong] [FEATURE] kyuubi-beeline supports --conf. Authored-by: Bruce Wong <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
…n page # 🔍 Description ## Issue References 🔗 [Good First Issues](https://kyuubi.readthedocs.io/en/master/contributing/code/get_started.html) image links to the Github issues page while the image on [Contributing Documentation](https://kyuubi.readthedocs.io/en/master/contributing/doc/get_started.html) does not. ## Describe Your Solution 🔧 Added a link to https://github.com/apache/kyuubi/issues?q=is%3Aopen+is%3Aissue+label%3Akind%3Adocumentation to match the behaviour from good first issues ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6514 from artilexx/patch-1. Closes #6514 3d54ce9 [artilexx] Add link to github issues on documentation issues image Lead-authored-by: artilexx <[email protected]> Co-authored-by: artilexx <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
# 🔍 Description Canonicalize the words, and enrich the description for KSCH. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Review. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6512 from pan3793/kshc-doc. Closes #6512 201c113 [Cheng Pan] nit 1becc1e [Cheng Pan] nit 8d48c7c [Cheng Pan] fix aea1e03 [Cheng Pan] fix 5ba5094 [Cheng Pan] fix 0c40de4 [Cheng Pan] fix 63dd21d [Cheng Pan] nit 1be2661 [Cheng Pan] Improve docs for KSHC Authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
# 🔍 Description ## Issue References 🔗 This pull request fixes #6508 ## Describe Your Solution 🔧 Add the key-value pairs in optimizedConf to session conf，to make the Kyuubi.env.SPARK_HOME take effect ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ When I configure the following parameters in kyuubi-env.sh  Then submit a spark batch task through the restful interface and pass the kyuubi.engineEnv.SPARK_HOME parameter to change the default value ```curl -H "Content-Type: application/json" -X POST -d '{"batchType": "SPARK", "resource":"xxxx/spark-examples_2.12-3.3.2.jar", "name": "Spark-PI", "conf": {"spark.master":"yarn","hive.server2.proxy.user":"XXXX","kyuubi.engineEnv.SPARK_HOME":"XXXXX/cluster114/spark","kyuubi.engineEnv.HADOOP_CONF_DIR":"XXXXX/conf"}, "args": [10],"className": "org.apache.spark.examples.SparkPi"}' http://XXXXX:XXXX/api/v1/batches``` Observe the log and find that SPARK_HOME is not set and takes effect. It still uses the value in kyuubi-env.sh.  #### Behavior With This Pull Request 🎉  After this PR, the update was successful #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6510 from jiaoqingbo/6508. Closes #6508 e89268e [jiaoqingbo] [KYUUBI #6508] Add the key-value pairs in optimizedConf to session conf Authored-by: jiaoqingbo <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
# 🔍 Description ## Issue References 🔗 ## Describe Your Solution 🔧 I'd like to introduce the feature that allows users to forcibly kill an engine through API. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6008 from zhaohehuhu/dev-0123. Closes #6008 00c208a [Cheng Pan] fix 8721a2d [Cheng Pan] log efc7587 [Cheng Pan] client cd5129d [Cheng Pan] fix ut 5e1b6a1 [Cheng Pan] Update kyuubi-server/src/test/scala/org/apache/kyuubi/server/api/v1/AdminResourceSuite.scala 72d7df3 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/ApplicationOperation.scala 6d5d087 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/ApplicationOperation.scala b013194 [zhaohehuhu] move the position of log 0cdeede [zhaohehuhu] restore ENGINE_SPARK_REGISTER_ATTRIBUTES f826d05 [zhaohehuhu] reformat a13466e [zhaohehuhu] update doc and log string encoded 3a2f597 [zhaohehuhu] refactor ae24ea7 [zhaohehuhu] refactor UT 936a54e [Wang, Fei] register app mgr info 9bacc2c [hezhao2] fix UTs 11106d7 [Wang, Fei] comments ba57c2c [hezhao2] refactor code to delete the node and then kill application 634ceb6 [hezhao2] reformat ab31382 [hezhao2] reformat 513bcdc [hezhao2] fix UT 5062206 [hezhao2] get refId by user, sharelevel and subdomain 3ad9577 [hezhao2] rename params to support multiple engines 632c56b [hezhao2] fix unused import bd7bb45 [hezhao2] refactor fb9b251 [hezhao2] add default value for forceKill param 070aad0 [hezhao2] refactor 51827ec [hezhao2] fix UT f11e765 [hezhao2] add an UT 8a65cf1 [hezhao2] refactor code d6f82ff [hezhao2] refactor code f3ab9c5 [hezhao2] new parameter added to decide whether to kill forcefully handle the result of killApplication 5faa5b5 [hezhao2] kill engine forcibly Lead-authored-by: hezhao2 <[email protected]> Co-authored-by: zhaohehuhu <[email protected]> Co-authored-by: Cheng Pan <[email protected]> Co-authored-by: Cheng Pan <[email protected]> Co-authored-by: Wang, Fei <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
# 🔍 Description ## Issue References 🔗 This pull request fixes #6516 ## Describe Your Solution 🔧 1. Using `buildStaticChecked` instead of `build` for static method `fromUri` 2. Using `Array.empty[Object]` instead of empty argument when invoking the build method ## Types of changes 🔖 - [X] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ Throw error like when running the unit test ``` java.lang.RuntimeException at org.apache.kyuubi.util.reflect.DynMethods$UnboundMethod.invoke(DynMethods.java:80) at org.apache.kyuubi.engine.spark.KyuubiSparkUtil$.buildURI(KyuubiSparkUtil.scala:143) at org.apache.kyuubi.engine.spark.KyuubiSparkUtilSuite.$anonfun$new$1(KyuubiSparkUtilSuite.scala:33) at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85) at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83) at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) at org.scalatest.Transformer.apply(Transformer.scala:22) at org.scalatest.Transformer.apply(Transformer.scala:20) at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226) at org.scalatest.TestSuite.withFixture(TestSuite.scala:196) at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195) at org.scalatest.funsuite.AnyFunSuite.withFixture(AnyFunSuite.scala:1564) at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224) at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236) at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306) at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236) at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218) at org.scalatest.funsuite.AnyFunSuite.runTest(AnyFunSuite.scala:1564) at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269) at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413) at scala.collection.immutable.List.foreach(List.scala:431) at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401) at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396) at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475) at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269) at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268) at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564) at org.scalatest.Suite.run(Suite.scala:1114) at org.scalatest.Suite.run$(Suite.scala:1096) at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564) at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273) at org.scalatest.SuperEngine.runImpl(Engine.scala:535) at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273) at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272) at org.scalatest.funsuite.AnyFunSuite.run(AnyFunSuite.scala:1564) at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:47) at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13(Runner.scala:1321) at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13$adapted(Runner.scala:1315) at scala.collection.immutable.List.foreach(List.scala:431) at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1315) at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24(Runner.scala:992) at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24$adapted(Runner.scala:970) at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1481) at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:970) at org.scalatest.tools.Runner$.run(Runner.scala:798) at org.scalatest.tools.Runner.run(Runner.scala) at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2or3(ScalaTestRunner.java:43) at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:26) ``` #### Behavior With This Pull Request 🎉 Test passed #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6517 from jeanlyn/issue-6516. Closes #6516 c6bf8b6 [jeanlyn] adjust e50e585 [jeanlyn] adjust 8afdc0c [jeanlyn] fix a61ae70 [jeanlyn] fix buildURI error and adding unit tests Authored-by: jeanlyn <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot]
Can you help keep this open source service alive? 💖 Please sponsor : )