Skip to content

Commit

Permalink
[KYUUBI #5807] Simple norm the use of `kyuubi.operation.result.arrow.…
Browse files Browse the repository at this point in the history
…timestampAsString` and `kyuubi.operation.result.format`

# 🔍 Description
## Issue References 🔗

This pull request norm the arrow conf.

## Describe Your Solution 🔧

Using `kyuubi.operation.result.arrow.timestampAsString` and `kyuubi.operation.result.format` string to get conf . To replace as `ARROW_BASED_ROWSET_TIMESTAMP_AS_STRING.key` and `OPERATION_RESULT_FORMAT`. Make it more norm.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [x] Other

## Test Plan 🧪

No need test

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested

**Be nice. Be informative.**

Closes #5807 from zzzzming95/norm-conf.

Closes #5807

9e65bfa [佘志铭] norm the arrow conf

Authored-by: 佘志铭 <[email protected]>
Signed-off-by: liangbowen <[email protected]>
  • Loading branch information
zzzzming95 authored and bowenliang123 committed Dec 3, 2023
1 parent 13af6ae commit 88c8cc3
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ import org.apache.spark.sql.types.StructType

import org.apache.kyuubi.{KyuubiSQLException, Utils}
import org.apache.kyuubi.config.KyuubiConf
import org.apache.kyuubi.config.KyuubiConf.{OPERATION_SPARK_LISTENER_ENABLED, SESSION_PROGRESS_ENABLE, SESSION_USER_SIGN_ENABLED}
import org.apache.kyuubi.config.KyuubiConf.{ARROW_BASED_ROWSET_TIMESTAMP_AS_STRING, OPERATION_SPARK_LISTENER_ENABLED, SESSION_PROGRESS_ENABLE, SESSION_USER_SIGN_ENABLED}
import org.apache.kyuubi.config.KyuubiReservedKeys.{KYUUBI_SESSION_SIGN_PUBLICKEY, KYUUBI_SESSION_USER_KEY, KYUUBI_SESSION_USER_SIGN, KYUUBI_STATEMENT_ID_KEY}
import org.apache.kyuubi.engine.spark.KyuubiSparkUtil.{getSessionConf, SPARK_SCHEDULER_POOL_KEY}
import org.apache.kyuubi.engine.spark.events.SparkOperationEvent
Expand Down Expand Up @@ -271,7 +271,7 @@ abstract class SparkOperation(session: Session)
protected def resultFormat: String = "thrift"

protected def timestampAsString: Boolean = {
spark.conf.get("kyuubi.operation.result.arrow.timestampAsString", "false").toBoolean
spark.conf.get(ARROW_BASED_ROWSET_TIMESTAMP_AS_STRING.key, "false").toBoolean
}

protected def setSessionUserSign(): Unit = {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ class SparkSQLOperationManager private (name: String) extends OperationManager(n
val incrementalCollect = spark.conf.getOption(OPERATION_INCREMENTAL_COLLECT.key)
.map(_.toBoolean).getOrElse(operationIncrementalCollectDefault)
// TODO: respect the config of the operation ExecuteStatement, if it was set.
val resultFormat = spark.conf.get("kyuubi.operation.result.format", "thrift")
val resultFormat = spark.conf.get(OPERATION_RESULT_FORMAT.key, "thrift")
resultFormat.toLowerCase match {
case "arrow" =>
new ArrowBasedExecuteStatement(
Expand Down

0 comments on commit 88c8cc3

Please sign in to comment.