[Umbrella] Fork beeline module and cut out Hive deps #6146
Comments
4 tasks
pan3793
added a commit
that referenced
this issue
Mar 11, 2024
# 🔍 Description ## Issue References 🔗 This is the first step of #6146, to gain a clear commit history, this PR just simply copied the `hive-beeline` module from Apache Hive 3.1.3, with minimal change to pass the tests and manually test basic functionalities, following PRs are going to remove other Hive deps gradually. ## Describe Your Solution 🔧 - Copy source code and test case from Apache Hive 3.1.3 - Drop `org.apache.hive:hive-beeline:3.1.3` - Backport HIVE-21584 to support JDK 9+ - Drop `HiveCli`, `HiveSchemaTool` and `BeelineInPlaceUpdateStream`, and the corresponding test cases - Temporary ignore(will fix later) `TestClientCommandHookFactory#testConnectHook` because of error `NoClassDefFound org/apache/curator/RetryPolicy` - Tune testing code to pass UT ## Types of changes :bookmark: - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Minimal changes to pass the unit tests. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6109 from pan3793/fork-beeline. Closes #6109 885f9fe [Cheng Pan] NOTICE a2efa1c [Cheng Pan] fix 5bb1cc9 [Cheng Pan] Copy from Apache Hive 3.1.3 Authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
4 tasks
zhouyifan279
pushed a commit
to zhouyifan279/kyuubi
that referenced
this issue
Mar 11, 2024
# 🔍 Description ## Issue References 🔗 This is the first step of apache#6146, to gain a clear commit history, this PR just simply copied the `hive-beeline` module from Apache Hive 3.1.3, with minimal change to pass the tests and manually test basic functionalities, following PRs are going to remove other Hive deps gradually. ## Describe Your Solution 🔧 - Copy source code and test case from Apache Hive 3.1.3 - Drop `org.apache.hive:hive-beeline:3.1.3` - Backport HIVE-21584 to support JDK 9+ - Drop `HiveCli`, `HiveSchemaTool` and `BeelineInPlaceUpdateStream`, and the corresponding test cases - Temporary ignore(will fix later) `TestClientCommandHookFactory#testConnectHook` because of error `NoClassDefFound org/apache/curator/RetryPolicy` - Tune testing code to pass UT ## Types of changes :bookmark: - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Minimal changes to pass the unit tests. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes apache#6109 from pan3793/fork-beeline. Closes apache#6109 885f9fe [Cheng Pan] NOTICE a2efa1c [Cheng Pan] fix 5bb1cc9 [Cheng Pan] Copy from Apache Hive 3.1.3 Authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
pan3793
added a commit
that referenced
this issue
Mar 11, 2024
# 🔍 Description ## Issue References 🔗 This is the next step of #6146, cutting out most Hive deps(excepting `hive-common`) and recovering the skipped tests via minor code tunning. ## Describe Your Solution 🔧 - Drop `hive-jdbc`, `hive-service`, `hive-service-rpc` deps in the beeline module. - Migrate from `commons-lang` to `commons-lang3` in the beeline module. - Recover the skipped test `TestClientCommandHookFactory#connectHook` ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Pass GA, and manually test to ensure the following error has gone. Before ``` roothadoop-master1:/opt/kyuubi# bin/beeline --version Warn: Not find kyuubi environment file /etc/kyuubi/conf/kyuubi-env.sh, using default ones... java.lang.NoClassDefFoundError: org/apache/curator/RetryPolicy ``` After ``` roothadoop-master1:/opt/kyuubi# bin/beeline --version Connecting to jdbc:hive2://hadoop-master1.orb.local:10000/default;password=hive;user=hive Connected to: Apache Hive (version 2.3.9) Driver: Kyuubi Project Hive JDBC Client (version 1.9.0-SNAPSHOT) Beeline version 1.9.0-SNAPSHOT by Apache Kyuubi 0: jdbc:hive2://hadoop-master1.orb.local:1000> ``` --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6153 from pan3793/beeline-2. Closes #6153 8cd52e5 [Cheng Pan] notice d03c729 [Cheng Pan] minor 5d16bf4 [Cheng Pan] beeline test pass Authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
zhaohehuhu
pushed a commit
to zhaohehuhu/incubator-kyuubi
that referenced
this issue
Mar 21, 2024
# 🔍 Description ## Issue References 🔗 This is the first step of apache#6146, to gain a clear commit history, this PR just simply copied the `hive-beeline` module from Apache Hive 3.1.3, with minimal change to pass the tests and manually test basic functionalities, following PRs are going to remove other Hive deps gradually. ## Describe Your Solution 🔧 - Copy source code and test case from Apache Hive 3.1.3 - Drop `org.apache.hive:hive-beeline:3.1.3` - Backport HIVE-21584 to support JDK 9+ - Drop `HiveCli`, `HiveSchemaTool` and `BeelineInPlaceUpdateStream`, and the corresponding test cases - Temporary ignore(will fix later) `TestClientCommandHookFactory#testConnectHook` because of error `NoClassDefFound org/apache/curator/RetryPolicy` - Tune testing code to pass UT ## Types of changes :bookmark: - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Minimal changes to pass the unit tests. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes apache#6109 from pan3793/fork-beeline. Closes apache#6109 885f9fe [Cheng Pan] NOTICE a2efa1c [Cheng Pan] fix 5bb1cc9 [Cheng Pan] Copy from Apache Hive 3.1.3 Authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
zhaohehuhu
pushed a commit
to zhaohehuhu/incubator-kyuubi
that referenced
this issue
Mar 21, 2024
…ne module # 🔍 Description ## Issue References 🔗 This is the next step of apache#6146, cutting out most Hive deps(excepting `hive-common`) and recovering the skipped tests via minor code tunning. ## Describe Your Solution 🔧 - Drop `hive-jdbc`, `hive-service`, `hive-service-rpc` deps in the beeline module. - Migrate from `commons-lang` to `commons-lang3` in the beeline module. - Recover the skipped test `TestClientCommandHookFactory#connectHook` ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Pass GA, and manually test to ensure the following error has gone. Before ``` roothadoop-master1:/opt/kyuubi# bin/beeline --version Warn: Not find kyuubi environment file /etc/kyuubi/conf/kyuubi-env.sh, using default ones... java.lang.NoClassDefFoundError: org/apache/curator/RetryPolicy ``` After ``` roothadoop-master1:/opt/kyuubi# bin/beeline --version Connecting to jdbc:hive2://hadoop-master1.orb.local:10000/default;password=hive;user=hive Connected to: Apache Hive (version 2.3.9) Driver: Kyuubi Project Hive JDBC Client (version 1.9.0-SNAPSHOT) Beeline version 1.9.0-SNAPSHOT by Apache Kyuubi 0: jdbc:hive2://hadoop-master1.orb.local:1000> ``` --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes apache#6153 from pan3793/beeline-2. Closes apache#6153 8cd52e5 [Cheng Pan] notice d03c729 [Cheng Pan] minor 5d16bf4 [Cheng Pan] beeline test pass Authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Code of Conduct
Search before asking
Describe the proposal
Background
Currently, we extend
org.apache.hive:hive-beeline:3.1.3to implement Kyuubi-specific features, this way has certain drawbacks. Hive has horrible dependencies hell(and some involve CVEs), we have put a lot of effort previously into making the Hive deps manageable, while it still has a chance in corner cases that trigger class link issuesAnother example is #5918
proposal
Fork beeline module from Apache Hive 3.1.3, and drop dependency org.apache.hive:hive-beeline:3.1.3, then gradually cut Hive transitive deps step by step, like Spark did in https://github.com/apache/spark/tree/master/sql/hive-thriftserver/src/main/java/org/apache/hive/service
To be clear, we should do our best to minimize the change on the forked code, to make it easy to backport patches from upstream in the future.
BTW, we already get benefits in the Kyuubi Hive JDBC module by forking the code.
Task list
Are you willing to submit PR?
The text was updated successfully, but these errors were encountered: