[pull] master from apache:master #40
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![pull[bot]](https://avatars.githubusercontent.com/in/12910?s=60&v=4){kind=small}
# 🔍 Description Upgrade to latest Apache Iceberg 1.6.0 ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Pass existing GHA. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6558 from pan3793/iceberg-1.6.0. Closes #6558 75e4293 [Cheng Pan] nit ba508b6 [Cheng Pan] nit 0180521 [Cheng Pan] Bump Iceberg 1.6.0 Authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
pull
bot
added
⤵️ pull
and removed
kind:documentation
kind:build
module:kubernetes
labels
…et the 'database' attribute if it's a Paimon plan. # 🔍 Description ## Issue References 🔗 This pull request fixes #6541 ## Describe Your Solution 🔧 Fix an issue where DataSourceV2RelationTableExtractor#table could not fetch the ‘database’ attribute causing the Ranger checks to fail when using the Paimon Catalog. If the database attribute is not resolved, use DataSourceV2RelationTableExtractor#identifier to complete it. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6544 from promising-forever/issues/6541. Closes #6541 6549f85 [caoyu] Fix test failure, paimon-spark run on Scala 2.12. c1a0921 [caoyu] Optimising the 'database' capture logic 69fb0bc [caoyu] PolicyJsonFileGenerator#genPolicies add paimonNamespace c89c70b [caoyu] [KYUUBI #6541] [AUTHZ] Fix DataSourceV2RelationTableExtractor#table can't get the 'database' attribute if it's a Paimon plan. 77f121b [caoyu] [KYUUBI #6541] [AUTHZ] Fix DataSourceV2RelationTableExtractor#table can't get the 'database' attribute if it's a Paimon plan. 9cfb584 [caoyu] [KYUUBI #6541] [AUTHZ] Fix DataSourceV2RelationTableExtractor#table can't get the 'database' attribute if it's a Paimon plan. Authored-by: caoyu <[email protected]> Signed-off-by: Bowen Liang <[email protected]>
# 🔍 Description ## Issue References 🔗 fix typos ## Describe Your Solution 🔧 anderror -> an error deprected -> deprecated ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6566 from XorSum/fix/kyuubi-conf-typo. Closes #6566 5c270ad [xorsum] fix typo in KyuubiConf Authored-by: xorsum <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
# 🔍 Description ## Issue References 🔗 This pull request fixes #6567 ## Describe Your Solution 🔧 With `setuptools==72.0.0`, the `test` command was removed after being deprecated for a long time. This completely breaks the installation of any packages using the `test` command, including PyHive. This PR fixes PyHive to be compatible with `setuptools` 72+ again. It seems that `setup.py test` wasn't used anywhere at all, which is why I just removed it. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ ``` $ pip install -e . Obtaining file:///home/localstack/Repos/kyuubi/python Preparing metadata (setup.py) ... error error: subprocess-exited-with-error × python setup.py egg_info did not run successfully. │ exit code: 1 ╰─> [6 lines of output] Traceback (most recent call last): File "<string>", line 2, in <module> File "<pip-setuptools-caller>", line 34, in <module> File "/home/localstack/Repos/kyuubi/python/setup.py", line 4, in <module> from setuptools.command.test import test as TestCommand ModuleNotFoundError: No module named 'setuptools.command.test' [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed × Encountered error while generating package metadata. ╰─> See above for output. note: This is an issue with the package mentioned above, not pip. hint: See above for details ``` #### Behavior With This Pull Request 🎉 ``` $ pip install --upgrade setuptools Requirement already satisfied: setuptools in .../.pyenv/versions/3.11.5/lib/python3.11/site-packages (72.0.0) $ pip install -e . Obtaining file:///home/.../repos/kyuubi/python Preparing metadata (setup.py) ... done Collecting future (from PyHive==0.7.0) Obtaining dependency information for future from https://files.pythonhosted.org/packages/da/71/ae30dadffc90b9006d77af76b393cb9dfbfc9629f339fc1574a1c52e6806/future-1.0.0-py3-none-any.whl.metadata Using cached future-1.0.0-py3-none-any.whl.metadata (4.0 kB) Requirement already satisfied: python-dateutil in /home/.../.pyenv/versions/3.11.5/lib/python3.11/site-packages (from PyHive==0.7.0) (2.8.2) Requirement already satisfied: six>=1.5 in /home/.../.pyenv/versions/3.11.5/lib/python3.11/site-packages (from python-dateutil->PyHive==0.7.0) (1.16.0) Using cached future-1.0.0-py3-none-any.whl (491 kB) Installing collected packages: future, PyHive Running setup.py develop for PyHive Successfully installed PyHive-0.7.0 future-1.0.0 [notice] A new release of pip is available: 23.2.1 -> 24.1.2 [notice] To update, run: pip install --upgrade pip ``` #### Related Unit Tests None --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6568 from alexrashed/fix-pyhive-setuptools-72. Closes #6567 acfb809 [Alexander Rashed] remove usage of setuptools.command.test Authored-by: Alexander Rashed <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
# 🔍 Description ## Issue References 🔗 This pull request fixes # ## Describe Your Solution 🔧 - bump `scala-maven-plugin` from 4.8.0 to 4.9.2, with `zinc` bumped from 1.8.0(Nov 11, 2022) to 1.10.0(May 6, 2024) ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6364 from bowenliang123/scala-maven-490. Closes #6364 5543797 [Bowen Liang] bump scala-maven-plugin to 4.9.2 Authored-by: Bowen Liang <[email protected]> Signed-off-by: Bowen Liang <[email protected]>
# 🔍 Description ## Issue References 🔗 This pr enhance the kyuubi batch logs for spark on k8s with kyuubi-ctl. ## Describe Your Solution 🔧 For spark on k8s, before, it does not show the app detail info until the log batch command finished. In this PR, it will show the batch app details in app state change. Especially show the app URL info. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests <img width="1326" alt="image" src="https://github.com/user-attachments/assets/2356af83-4422-4dfc-812c-d90b809ea724"> --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6579 from turboFei/print_batch_report. Closes #6579 0651dad [Wang, Fei] show app details on app state change Authored-by: Wang, Fei <[email protected]> Signed-off-by: Wang, Fei <[email protected]>
[KYUUBI #6549] Fix 'Could not find or load main class when launching engine' # 🔍 Description ## Issue References 🔗 This pull request fixes #6549 ## Describe Your Solution 🔧 When obtaining configuration items, if it is null or empty, return none ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6556 from LiJie20190102/launch_engine. Closes #6549 c57a08a [lijie0203] [KYUUBI #6549] Fix 'Could not find or load main class when launching engine' 642d807 [lijie0203] [KYUUBI #6549] Fix 'Could not find or load main class when launching engine' 6792609 [lijie0203] [KYUUBI #6549] Fix 'Could not find or load main class when launching engine' 4ba9fb5 [lijie0203] [KYUUBI #6549] Fix 'Could not find or load main class when launching engine' Authored-by: lijie0203 <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
… Engine # 🔍 Description ## Issue References 🔗 This pull request fixes #6574 ## Describe Your Solution 🔧 Skip eagerly execute command in physical and execution plan only mode ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [X] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Related Unit Tests added unit test --- # Checklist 📝 - [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6575 from wForget/KYUUBI-6574. Closes #6574 6f79228 [wforget] fix 9aff4a8 [wforget] fix 839ea4a [wforget] fix 8a08c9f [wforget] [KYUUBI #6574] Skip eagerly execute command in PlanOnly mode of Spark Engine Authored-by: wforget <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
# 🔍 Description ## Issue References 🔗 This pull request fixes #6564 ## Describe Your Solution 🔧 Remove the `columnDesc` for `InsertIntoHadoopFsRelationCommand ` and `InsertIntoHiveTable ` in `table_command_spec.json` ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [x] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ Insert into table will check the privilege of columns. #### Behavior With This Pull Request 🎉 Insert into table will check the privilege of table. #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6570 from liujiayi771/insert-permission. Closes #6564 d956aa9 [joey.ljy] Fix ut d282f8e [joey.ljy] insert into table check the privilege of table Authored-by: joey.ljy <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
# 🔍 Description Flink 1.20 is out, [Release Note](https://nightlies.apache.org/flink/flink-docs-release-1.20/release-notes/flink-1.20/) ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 CI is updated to cover Flink 1.20 --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6557 from pan3793/flink-1.20. Closes #6557 a414094 [Cheng Pan] remove rc 8ee4cf5 [Cheng Pan] fix url fbaf660 [Cheng Pan] docs ddbd10f [Cheng Pan] Support Flink 1.20 Authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
…ecial characters # 🔍 Description ## Issue References 🔗 This pull request fixes #6581 ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. I modified `KyuubiSparkSQLAstBuilder#visitMultipartIdentifier` and implemented `KyuubiSparkSQLAstBuilder#visitQuotedIdentifier` to process the quoted identifiers. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests ``` extensions/spark/kyuubi-extension-spark-3-3/src/test/scala/org/apache/spark/sql/ZorderSuiteBase.scala test("optimize sort by backquoted column name") ``` --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6582 from XorSum/features/zorder-backquote. Closes #6582 16ffa12 [xorsum] zorder by support quote Authored-by: xorsum <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
# 🔍 Description ## Issue References 🔗 This pull request fixes # ## Describe Your Solution 🔧 - Bump commons-io from 2.11.0(2021-07-09) to 2.16.1(2024-04-04) for accumulate updates, release note: https://commons.apache.org/proper/commons-io/changes-report.html#a2.16.1 ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6586 from bowenliang123/commonsio-216. Closes #6586 4fbb056 [Bowen Liang] bump commons-io to 2.16.1 Authored-by: Bowen Liang <[email protected]> Signed-off-by: Bowen Liang <[email protected]>
# 🔍 Description ## Issue References 🔗 This pull request fixes #6565 ## Describe Your Solution 🔧 - Fixed misleading `kyuubi.metrics` properties in `charts/kyuubi/templates/kyuubi-configmap.yaml`. - Fixed condition to create `PodMonitor`, `ServiceMonitor` and `PrometheusRule`. - Move metrics related properties to `metrics` property tree. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [x] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Disabled metrics - prometheus port is not renderred ```shell helm template kyuubi charts/kyuubi --set metrics.enabled=false \ -s templates/kyuubi-statefulset.yaml \ -s templates/kyuubi-headless-service.yaml ``` #### JMX reporter only - prometheus port is not renderred ```shell helm template kyuubi charts/kyuubi --set metrics.reporters="JMX" \ -s templates/kyuubi-statefulset.yaml \ -s templates/kyuubi-headless-service.yaml ``` #### Default properties - prometheus port is renderred in StatefulSet and Headless Service ```shell helm template kyuubi charts/kyuubi -s templates/kyuubi-statefulset.yaml \ -s templates/kyuubi-headless-service.yaml ``` #### Default properties - PodMonitor is not renderred ```shell helm template kyuubi charts/kyuubi -s templates/kyuubi-podmonitor.yaml ``` #### Default properties - ServiceMonitor is not renderred ```shell helm template kyuubi charts/kyuubi -s templates/kyuubi-servicemonitor.yaml ``` #### Default properties - PrometheusRule is not renderred ```shell helm template kyuubi charts/kyuubi -s templates/kyuubi-alert.yaml ``` #### Enabled metrics, PodMonitor, ServiceMonitor, PrometheusRule, PROMETHEUS reporter and port set to 9999 ```shell helm template kyuubi charts/kyuubi --set metrics.enabled=true \ --set metrics.reporters="PROMETHEUS\, JMX" \ --set metrics.prometheusPort=9999 \ --set metrics.podMonitor.enabled=true \ --set metrics.serviceMonitor.enabled=true \ --set metrics.prometheusRule.enabled=true \ -s templates/kyuubi-statefulset.yaml \ -s templates/kyuubi-headless-service.yaml \ -s templates/kyuubi-configmap.yaml \ -s templates/kyuubi-podmonitor.yaml \ -s templates/kyuubi-servicemonitor.yaml \ -s templates/kyuubi-alert.yaml ``` #### Install the chart and test Prometheus endpoint ```shell helm install kyuubi charts/kyuubi --set metrics.enabled=true \ --set metrics.reporters="PROMETHEUS\, JMX" \ --set metrics.prometheusPort=9999 ... kyuubikyuubi-0:/opt/kyuubi$ curl 127.0.0.1:9999/metrics # HELP kyuubi_buffer_pool_mapped_count Generated from Dropwizard metric import (metric=kyuubi.buffer_pool.mapped.count, type=com.codahale.metrics.jvm.JmxAttributeGauge) # TYPE kyuubi_buffer_pool_mapped_count gauge kyuubi_buffer_pool_mapped_count 0.0 # HELP kyuubi_gc_MarkSweepCompact_time Generated from Dropwizard metric import (metric=kyuubi.gc.MarkSweepCompact.time, type=com.codahale.metrics.jvm.GarbageCollectorMetricSet$$Lambda$227/1493158871) # TYPE kyuubi_gc_MarkSweepCompact_time gauge kyuubi_gc_MarkSweepCompact_time 91.0 ... ``` --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6580 from dnskr/helm-improve-monitoring-configuration. Closes #6580 1f20cab [dnskr] [K8S][HELM] Improve metrics configuration Authored-by: dnskr <[email protected]> Signed-off-by: dnskr <[email protected]>
# 🔍 Description ## Issue References 🔗 This pull request fixes # ## Describe Your Solution 🔧 - bump apache parent pom from 31 to 33: - 32: https://github.com/apache/maven-apache-parent/releases/tag/apache-32 - [[MPOM-264](https://issues.apache.org/jira/browse/MPOM-264)] - Parameterize maven-compiler-plugin with parameter "release" when running on JDK 9+ - 33: https://github.com/apache/maven-apache-parent/releases/tag/apache-33 ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6571 from bowenliang123/mvn-parent-33. Closes #6571 bb367ef [Bowen Liang] bump parent pom to 33 Authored-by: Bowen Liang <[email protected]> Signed-off-by: Bowen Liang <[email protected]>
…atch jobs via REST API # 🔍 Description ## Issue References 🔗 ## Describe Your Solution 🔧 - support creating batch jobs with uploading extra resource files - allow uploading extra resource when creating batch jobs via REST API - support binding the subresources to configs by customed configs, eg.`spark.submit.pyFiles`. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests + new test --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6335 from bowenliang123/batch-subresource. Closes #6335 57d43d2 [Bowen Liang] nit d866a8a [Bowen Liang] warn exception 20d4328 [Bowen Liang] log exception when exception ignored 58c4023 [Bowen Liang] rename param to ignoreException 80bc210 [Bowen Liang] cleanup the uploaded resource folder when handling files error 3e79611 [Bowen Liang] throw exception when file non-existed 09ac48a [liangbowen] pyspark extra resources Lead-authored-by: Bowen Liang <[email protected]> Co-authored-by: liangbowen <[email protected]> Signed-off-by: Bowen Liang <[email protected]>
# 🔍 Description In some cases, env var USER is unavailable, for example, docker container, we should evaluate it via `id -nu` ref: https://github.com/apache/hadoop/blob/rel/release-3.4.0/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh#L893-L896 ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 ``` $ docker run -t -i ubuntu:latest root1dbeaefd6cd4:/# echo $USER root1dbeaefd6cd4:/# id -nu root root1dbeaefd6cd4:/# exit ``` --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6588 from pan3793/USER. Closes #6588 a797863 [Cheng Pan] Get effective USER if env var is unavailable Authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
… skip check for resource uploaded use case # 🔍 Description ## Issue References 🔗 As title, for security concern. Before, it skips to check the local dir access for resource upload use case, which is not expected. ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6592 from turboFei/check_batch_access. Closes #6592 80cbc74 [Wang, Fei] do not check if is empty a82fe8e [Wang, Fei] check batch access Authored-by: Wang, Fei <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
# 🔍 Description ## Issue References 🔗 Support authorization on swagger UI. ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6591 from turboFei/swagger_auth. Closes #6591 28010bd [Wang, Fei] Support authorization on swagger UI Authored-by: Wang, Fei <[email protected]> Signed-off-by: Wang, Fei <[email protected]>
Fixing some typos ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6596 from hyperbolic2346/hyperbolic2346-patch-1. Closes #6596 ebbddf0 [Mike Wilson] Update architecture.md Lead-authored-by: Mike Wilson <[email protected]> Co-authored-by: Mike Wilson <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
# 🔍 Description This PR disables Javadoc/Scaladoc by default, while still enabling that on Nexus deploy and CI, to speed up the normal building by developers daily performed. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Apple M1 Max, Java 17 ``` $ time build/mvn clean install -DskipTests ``` before: 1145.67s user 62.41s system 276% cpu 7:16.56 total after: 530.33s user 35.82s system 214% cpu 4:23.69 total --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6599 from pan3793/javadoc. Closes #6599 660f8dc [Cheng Pan] Disable Javadoc/Scaladoc by default Authored-by: Cheng Pan <[email protected]> Signed-off-by: liangbowen <[email protected]>
# 🔍 Description This PR makes `kyuubi-flink-sql-engine` compile success with Scala 2.13. Note: As of Flink 1.20, it does not support Scala 2.13, so won't expect the Flink engine to work with Scala 2.13 for now. It would be helpful in the future after Flink removes Scala dependencies(planed in 2.0) then we can use any version of Scala. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 ``` $ build/mvn clean install -DskipTests -Pscala-2.13 ... [INFO] Reactor Summary for Kyuubi Project Parent 1.10.0-SNAPSHOT: [INFO] [INFO] Kyuubi Project Parent .............................. SUCCESS [ 9.031 s] [INFO] Kyuubi Project Util ................................ SUCCESS [ 3.998 s] [INFO] Kyuubi Project Util Scala .......................... SUCCESS [ 8.579 s] [INFO] Kyuubi Project Common .............................. SUCCESS [ 26.006 s] [INFO] Kyuubi Project Embedded Zookeeper .................. SUCCESS [ 6.573 s] [INFO] Kyuubi Project High Availability ................... SUCCESS [ 12.360 s] [INFO] Kyuubi Project Rest Client ......................... SUCCESS [ 5.799 s] [INFO] Kyuubi Project Control ............................. SUCCESS [ 12.345 s] [INFO] Kyuubi Project Events .............................. SUCCESS [ 11.447 s] [INFO] Kyuubi Dev Spark Lineage Extension ................. SUCCESS [ 13.327 s] [INFO] Kyuubi Project Metrics ............................. SUCCESS [ 7.326 s] [INFO] Kyuubi Project Hive JDBC Client .................... SUCCESS [ 7.492 s] [INFO] Kyuubi Project Server Plugin ....................... SUCCESS [ 1.428 s] [INFO] Kyuubi Project Download Externals .................. SUCCESS [ 7.132 s] [INFO] Kyuubi Project Engine Spark SQL .................... SUCCESS [01:11 min] [INFO] Kyuubi Project Server .............................. SUCCESS [ 35.930 s] [INFO] Kyuubi Project Hive Beeline ........................ SUCCESS [ 6.833 s] [INFO] Kyuubi Spark Connector Common ...................... SUCCESS [ 10.399 s] [INFO] Kyuubi Spark TPC-DS Connector ...................... SUCCESS [ 13.854 s] [INFO] Kyuubi Spark TPC-H Connector ....................... SUCCESS [ 10.407 s] [INFO] Kyuubi Dev Code Coverage ........................... SUCCESS [ 1.701 s] [INFO] Kyuubi Spark JDBC Dialect plugin ................... SUCCESS [ 6.881 s] [INFO] Kyuubi Dev Spark Authorization Extension ........... SUCCESS [ 21.508 s] [INFO] Kyuubi Dev Spark Authorization Extension Shaded .... SUCCESS [ 0.627 s] [INFO] Kyuubi Project Engine Chat ......................... SUCCESS [ 10.577 s] [INFO] Kyuubi Project Engine Flink SQL .................... SUCCESS [ 22.605 s] [INFO] Kyuubi Project Engine Hive SQL ..................... SUCCESS [ 17.021 s] [INFO] Kyuubi Project Engine JDBC ......................... SUCCESS [ 13.871 s] [INFO] Kyuubi Project Engine Trino ........................ SUCCESS [ 14.339 s] [INFO] Kyuubi Test Integration Tests ...................... SUCCESS [ 0.122 s] [INFO] Kyuubi Test Flink SQL IT ........................... SUCCESS [ 4.563 s] [INFO] Kyuubi Test Hive IT ................................ SUCCESS [ 4.123 s] [INFO] Kyuubi Test Trino IT ............................... SUCCESS [ 3.992 s] [INFO] Kyuubi Project Hive JDBC Shaded Client ............. SUCCESS [ 10.347 s] [INFO] Kyuubi Test Jdbc IT ................................ SUCCESS [ 4.597 s] [INFO] Kyuubi Test Zookeeper IT ........................... SUCCESS [ 2.907 s] [INFO] Kyuubi Project Assembly ............................ SUCCESS [ 1.346 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 07:03 min [INFO] Finished at: 2024-08-09T11:38:13+08:00 [INFO] ------------------------------------------------------------------------ ``` --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6598 from pan3793/flink-scala-213. Closes #6598 78fcd20 [Cheng Pan] Flink engine module supports building with Scala 2.13 Authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
# 🔍 Description It was wrong applied to all goals previously, which caused all `test-compile`s are skipped. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Let's monitor what CI happens --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6600 from pan3793/scaladoc. Closes #6600 86ea6fd [Cheng Pan] scala213 2a76a93 [Cheng Pan] Correct usage of maven.scaladoc.skip Authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
…r Spark 4.0 # 🔍 Description As title. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Pass GHA. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6601 from pan3793/delta-4.0. Closes #6601 711f2e0 [Cheng Pan] Bump Delta Lake 4.0.0rc1 and enable Delta Lake test for Spark 4.0 Authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
…fAdvisor # 🔍 Description ## Issue References 🔗 This pull request fixes # ## Describe Your Solution 🔧 The current FileSessionConfAdvisor only supports configuring a single file through kyuubi.session.conf.profiler. However, users may need to use multiple configuration files, and the existing setup does not support this. To address this need, we have updated the FileSessionConfAdvisor to support multiple configuration files. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6585 from lsm1/branch-file-advidor-support-chain. Closes #6585 15b6524 [senmiaoliu] fix style fcedc4f [senmiaoliu] FileSessionConfAdvisor support multi files Authored-by: senmiaoliu <[email protected]> Signed-off-by: senmiaoliu <[email protected]>
# 🔍 Description
I found sometimes `build/mvn -version` is pretty slow even `build/apache-maven-${MVN_VERSION}` is already cached, and I found it may stuck at the following call when network quality is not good.
https://github.com/apache/kyuubi/blob/bdd91f45396ab20b6dfe0a266fb8bfe340d99192/build/mvn#L86
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Tested `build/mvn -version` in the following cases:
1. when `build/apache-maven-${MVN_VERSION}/bin/mvn` is available, the command always finishes quickly.
2. when `build/apache-maven-${MVN_VERSION}/bin/mvn` is unavailable but global installed `mvn` version matches `maven.version` defined in root `pom.xml`, the command always finishes quickly too.
3. otherwise, it automatically downloads the maven binary tarball from network.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6606 from pan3793/mvn.
Closes #6606
a801d79 [Cheng Pan] nit
3a9f35f [Cheng Pan] build/mvn checks local downloaded binary ahead before network access
Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
…ed to match # 🔍 Description ## Issue References 🔗 This pull request fixes #6485 ## Describe Your Solution 🔧 Ignore uppercase and lowercase letters in table names when using regular expressions to match. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Added unit tests when table names have capital letters. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6605 from BruceWong96/fix-presto-regex. Closes #6485 06f737f [Bruce Wong] Fix typos 9307175 [Bruce Wong] Added unit tests for table names with both upper and lower case letters 9837030 [Bruce Wong] fix table not found Authored-by: Bruce Wong <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot]
Can you help keep this open source service alive? 💖 Please sponsor : )