[FEATURE][SPARK] Collect and summarize the executorRunTime and executorCpuTime of the statement
#6107
Closed
3 of 4 tasks
Assignees
Labels
Comments
4 tasks
cxzl25
changed the title
executorRunTime and executorCpuTime of the statement
|
please assign to me |
4 tasks
cxzl25
pushed a commit
that referenced
this issue
Mar 4, 2024
…of the Spark event # 🔍 Description ## Issue References 🔗 This pull request fixes #6112 (comment) ## Describe Your Solution 🔧 add comments for the newly added parameters ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6124 from XorSum/features/features/spark-engine-cpu-time-collect-comments. Closes #6107 d93028b [bkhan] add comments Authored-by: bkhan <[email protected]> Signed-off-by: Shaoyun Chen <[email protected]>
zhaohehuhu
pushed a commit
to zhaohehuhu/incubator-kyuubi
that referenced
this issue
Mar 21, 2024
…me` and `executorCpuTime` of the statement # 🔍 Description ## Issue References 🔗 This pull request fixes apache#6107 ## Describe Your Solution 🔧 The total execution time of a statement (or a session) is the summary of the execution time of the stages belonging to the statement (or session). The total execution time of a stage is collected from `SQLOperationListener#onStageCompleted`. The total execution times of the statement or a session are stored in the engine events or output to the log. <img width="962" alt="截屏2024-02-29 14 47 50" src="https://github.com/apache/kyuubi/assets/23011702/176df1db-bb20-428b-94b8-fa02c946fde2"> <img width="1143" alt="截屏2024-02-29 14 47 21" src="https://github.com/apache/kyuubi/assets/23011702/8cfc6a72-f6e8-45b6-bdda-30296c94c893"> ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes apache#6112 from XorSum/features/spark-engine-cpu-time-collect. Closes apache#6107 8028005 [bkhan] check same group d9efa2d [bkhan] formatDuration a8841cd [bkhan] update 2507159 [bkhan] Apply suggestions from code review cfed2b9 [bkhan] use formatDurationVerbose 444d4aa [bkhan] Collect and summarize the executorRunTime and executorCpuTime of the statement Authored-by: bkhan <[email protected]> Signed-off-by: Shaoyun Chen <[email protected]>
zhaohehuhu
pushed a commit
to zhaohehuhu/incubator-kyuubi
that referenced
this issue
Mar 21, 2024
…eters of the Spark event # 🔍 Description ## Issue References 🔗 This pull request fixes apache#6112 (comment) ## Describe Your Solution 🔧 add comments for the newly added parameters ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes apache#6124 from XorSum/features/features/spark-engine-cpu-time-collect-comments. Closes apache#6107 d93028b [bkhan] add comments Authored-by: bkhan <[email protected]> Signed-off-by: Shaoyun Chen <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Code of Conduct
Search before asking
Describe the feature
By aggregating the statement's
taskMetrics.executorRunTimeandtaskMetrics.executorCpuTimeto collect the CPU time consumed by the statement, a rough SQL-level billing can be achieved.Motivation
No response
Describe the solution
No response
Additional context
No response
Are you willing to submit PR?
The text was updated successfully, but these errors were encountered: